2024-04-15T22:08:58.645 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T22:08:58.645 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536 --name teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi --verbose --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536/orig.config.yaml 2024-04-15T22:08:58.682 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T22:08:58.790 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7657536' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 7775 sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 sleep_before_teardown: 0 subset: 18/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE14Iq7Kjy9N+JtU1nVMkjYzi9Mp+oObvzqyMXMm5pAF483WD5vkoPI+bKtF6+56igBtQv1YEAqD7elEdt/dT9g= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEQSrcVKz7rr8Aj1XdGsSz8z5fvtonxc+F+aC6iCDktwfN92oYzaGY7eoEwFhmoBG6OAeJiOuo3WnRMEMktgrU= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-15T22:08:58.791 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa; will attempt to use it 2024-04-15T22:08:58.792 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks 2024-04-15T22:08:58.792 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T22:08:58.794 INFO:teuthology.task.internal:Checking packages... 2024-04-15T22:08:58.828 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a9a752df26c63acad72e1b3569fd79a515ca0765' 2024-04-15T22:08:58.828 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T22:08:58.829 INFO:teuthology.packaging:ref: None 2024-04-15T22:08:58.829 INFO:teuthology.packaging:tag: None 2024-04-15T22:08:58.829 INFO:teuthology.packaging:branch: squid 2024-04-15T22:08:58.829 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:08:58.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-15T22:08:59.107 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2114.ga9a752df 2024-04-15T22:08:59.109 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T22:08:59.115 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T22:08:59.115 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T22:08:59.136 INFO:teuthology.task.internal:Saving configuration 2024-04-15T22:08:59.149 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T22:08:59.154 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T22:08:59.178 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:14.352992', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE14Iq7Kjy9N+JtU1nVMkjYzi9Mp+oObvzqyMXMm5pAF483WD5vkoPI+bKtF6+56igBtQv1YEAqD7elEdt/dT9g='} 2024-04-15T22:08:59.203 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:14.351911', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEQSrcVKz7rr8Aj1XdGsSz8z5fvtonxc+F+aC6iCDktwfN92oYzaGY7eoEwFhmoBG6OAeJiOuo3WnRMEMktgrU='} 2024-04-15T22:08:59.203 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T22:08:59.210 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-15T22:08:59.210 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-15T22:08:59.210 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T22:08:59.278 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7a37773f70>, signals=[15]) 2024-04-15T22:08:59.279 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T22:08:59.285 INFO:teuthology.task.internal:Opening connections... 2024-04-15T22:08:59.285 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:08:59.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:08:59.410 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-04-15T22:08:59.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:08:59.499 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T22:08:59.504 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-04-15T22:08:59.522 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-04-15T22:08:59.522 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-04-15T22:08:59.578 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T22:08:59.579 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T22:08:59.580 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-04-15T22:08:59.603 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-04-15T22:08:59.620 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-04-15T22:08:59.620 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-04-15T22:08:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-04-15T22:08:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-04-15T22:08:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-04-15T22:08:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-04-15T22:08:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-04-15T22:08:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-04-15T22:08:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T22:08:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-04-15T22:08:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-04-15T22:08:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T22:08:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-04-15T22:08:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T22:08:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T22:08:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T22:08:59.678 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-04-15T22:08:59.702 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T22:08:59.710 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T22:08:59.715 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T22:08:59.715 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-04-15T22:08:59.717 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-04-15T22:08:59.733 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T22:08:59.738 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T22:08:59.738 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-04-15T22:08:59.774 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-04-15T22:08:59.806 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T22:08:59.862 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T22:08:59.874 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-15T22:08:59.875 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-15T22:08:59.875 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T22:08:59.875 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T22:08:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:08:59.891 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-15T22:08:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:08:59.892 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-04-15T22:08:59.947 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-361.el9.x86_64 2024-04-15T22:08:59.947 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-361.el9.x86_64 2024-04-15T22:08:59.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-04-15T22:08:59.949 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-361.el9.x86_64 2024-04-15T22:08:59.949 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.14.0-361.el9.x86_64 2024-04-15T22:08:59.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-04-15T22:09:02.438 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-04-15T22:09:02.528 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-04-15T22:09:03.902 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-04-15T22:09:04.080 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-04-15T22:09:05.579 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-04-15T22:09:08.474 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-15T22:09:13.942 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 848 kB/s | 8.0 MB 00:09 2024-04-15T22:09:15.735 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-15T22:09:21.354 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 675 kB/s | 6.2 MB 00:09 2024-04-15T22:09:21.502 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.2 MB 00:02 2024-04-15T22:09:22.780 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-04-15T22:09:22.978 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-04-15T22:09:24.821 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-15T22:09:25.848 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-04-15T22:09:29.930 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-15T22:09:30.141 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:01 ago on Mon 15 Apr 2024 10:09:29 PM UTC. 2024-04-15T22:09:30.839 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-04-15T22:09:32.253 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T22:09:32.253 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T22:09:32.353 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-04-15T22:09:32.361 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-04-15T22:09:32.362 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:09:32.363 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 77 M 2024-04-15T22:09:32.363 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 123 M 2024-04-15T22:09:32.363 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-04-15T22:09:32.623 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.0 MB/s | 277 kB 00:00 2024-04-15T22:09:33.187 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T22:09:33.188 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T22:09:33.299 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-04-15T22:09:33.308 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-04-15T22:09:33.309 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:09:33.310 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 77 M 2024-04-15T22:09:33.310 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 123 M 2024-04-15T22:09:33.310 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-04-15T22:09:33.665 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 781 kB/s | 277 kB 00:00 2024-04-15T22:09:34.048 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 9.0 MB/s | 15 MB 00:01 2024-04-15T22:09:35.006 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 12 MB/s | 28 MB 00:02 2024-04-15T22:09:35.023 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 8.9 MB/s | 15 MB 00:01 2024-04-15T22:09:35.307 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 11 MB/s | 34 MB 00:02 2024-04-15T22:09:35.310 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:09:35.310 INFO:teuthology.orchestra.run.smithi081.stdout:Total 26 MB/s | 77 MB 00:02 2024-04-15T22:09:35.311 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-04-15T22:09:35.682 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-04-15T22:09:35.683 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-04-15T22:09:36.559 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-04-15T22:09:36.559 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-04-15T22:09:36.739 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 9.0 MB/s | 28 MB 00:03 2024-04-15T22:09:37.049 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 9.0 MB/s | 34 MB 00:03 2024-04-15T22:09:37.052 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:09:37.053 INFO:teuthology.orchestra.run.smithi163.stdout:Total 21 MB/s | 77 MB 00:03 2024-04-15T22:09:37.053 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-04-15T22:09:37.446 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-04-15T22:09:37.446 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-04-15T22:09:38.290 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-04-15T22:09:38.290 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-04-15T22:09:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-04-15T22:09:39.459 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:09:39.541 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:40.602 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-04-15T22:09:41.219 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:09:41.299 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:42.291 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:44.105 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:44.244 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:47.188 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:47.262 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:09:48.998 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:49.042 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:09:52.020 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:09:53.801 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:21.546 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:21.547 INFO:teuthology.orchestra.run.smithi081.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T22:10:21.547 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:10:21.589 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:21.986 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:21.986 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:10:21.986 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:10:21.987 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:10:22.393 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:22.393 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:10:22.393 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-04-15T22:10:22.394 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:22.394 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:22.394 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T22:10:22.394 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:22.394 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:10:22.394 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-04-15T22:10:22.733 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:22.833 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:22.833 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:22.833 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T22:10:22.833 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T22:10:22.833 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T22:10:22.833 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-15T22:10:22.833 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T22:10:22.834 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:22.914 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:22.914 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:22.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-04-15T22:10:22.948 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-04-15T22:10:22.948 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-04-15T22:10:22.948 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-04-15T22:10:22.949 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-04-15T22:10:22.950 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:10:22.950 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T22:10:22.950 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T22:10:22.950 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T22:10:22.950 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-04-15T22:10:22.950 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:10:22.950 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T22:10:22.951 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T22:10:22.951 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-438.el9.x86_64 2024-04-15T22:10:22.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T22:10:23.128 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:23.128 INFO:teuthology.orchestra.run.smithi163.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T22:10:23.128 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:10:23.171 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:23.535 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:23.535 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:10:23.535 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:10:23.535 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:10:23.560 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:10:23.943 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-04-15T22:10:24.286 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:24.375 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:24.375 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:24.375 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T22:10:24.376 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T22:10:24.376 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T22:10:24.376 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-15T22:10:24.376 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T22:10:24.376 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:24.456 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:24.456 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:24.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-04-15T22:10:24.490 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-04-15T22:10:24.490 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-04-15T22:10:24.490 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.06 2024-04-15T22:10:24.490 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 68.el9 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Group : Unspecified 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 8264254 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : builder@centos.org 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-04-15T22:10:24.491 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:10:24.492 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T22:10:24.492 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T22:10:24.492 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T22:10:24.492 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-04-15T22:10:24.492 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:10:24.492 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T22:10:24.493 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T22:10:24.493 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 5.14.0-438.el9.x86_64 2024-04-15T22:10:24.493 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T22:10:24.961 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T22:10:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-04-15T22:10:25.004 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-04-15T22:10:25.024 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.2Jlp4HNVyp 2024-04-15T22:10:25.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2Jlp4HNVyp 2024-04-15T22:10:25.069 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-04-15T22:10:25.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.2Jlp4HNVyp 2024-04-15T22:10:25.235 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.2Jlp4HNVyp is 6KB 2024-04-15T22:10:25.283 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.2Jlp4HNVyp 2024-04-15T22:10:25.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T22:10:25.363 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T22:10:25.363 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T22:10:25.363 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T22:10:25.363 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T22:10:25.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T22:10:25.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-04-15T22:10:26.421 INFO:teuthology.orchestra.run.smithi163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T22:10:26.447 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-04-15T22:10:26.449 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-04-15T22:10:26.464 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.3UYQoIvE1M 2024-04-15T22:10:26.464 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3UYQoIvE1M 2024-04-15T22:10:26.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.3UYQoIvE1M 2024-04-15T22:10:26.669 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.3UYQoIvE1M is 6KB 2024-04-15T22:10:26.717 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.3UYQoIvE1M 2024-04-15T22:10:26.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T22:10:26.795 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T22:10:26.795 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T22:10:26.796 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T22:10:26.796 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T22:10:26.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T22:10:26.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-04-15T22:10:55.546 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-15T22:10:55.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-15T22:10:55.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:10:56.953 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-15T22:10:56.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:10:56.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:14.079 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-15T22:11:15.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-04-15T22:11:23.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-15T22:11:23.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:24.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:11:24.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:26.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-15T22:11:27.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-04-15T22:11:38.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-15T22:11:38.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:39.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:11:39.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:42.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-04-15T22:11:56.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-15T22:11:57.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:11:57.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:12:11.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-15T22:12:11.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:12:12.026 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-15T22:12:12.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-15T22:12:12.445 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T22:12:12.445 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-15T22:12:12.460 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-438.el9.x86_64 2024-04-15T22:12:12.460 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T22:12:12.460 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T22:12:12.460 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T22:12:13.461 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-15T22:12:13.461 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T22:12:13.554 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-04-15T22:12:13.585 DEBUG:teuthology.parallel:result is None 2024-04-15T22:12:16.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-04-15T22:12:34.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:12:34.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:12:34.505 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-04-15T22:12:34.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:12:34.933 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T22:12:34.933 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-04-15T22:12:34.948 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-438.el9.x86_64 2024-04-15T22:12:34.948 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T22:12:34.948 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T22:12:34.949 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T22:12:35.949 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-15T22:12:35.950 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T22:12:36.005 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-04-15T22:12:36.041 DEBUG:teuthology.parallel:result is None 2024-04-15T22:12:36.041 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T22:12:36.048 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T22:12:36.049 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T22:12:36.052 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T22:12:36.069 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T22:12:36.076 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T22:12:36.083 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T22:12:36.083 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T22:12:36.113 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T22:12:36.144 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T22:12:36.151 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T22:12:36.151 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T22:12:36.183 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T22:12:36.216 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:36.217 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:36.230 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:36.231 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:36.233 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T22:12:36.240 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T22:12:36.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T22:12:36.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T22:12:36.305 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T22:12:36.315 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T22:12:36.315 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T22:12:36.348 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T22:12:36.363 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:36.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:36.497 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:36.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:36.624 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:12:36.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T22:12:36.693 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:36.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:36.798 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:36.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:36.883 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-04-15T22:12:36.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T22:12:36.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-04-15T22:12:36.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-04-15T22:12:37.015 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:12:37.046 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:12:37.342 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T22:12:37.351 INFO:teuthology.task.internal:Starting timer... 2024-04-15T22:12:37.351 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T22:12:37.362 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T22:12:37.373 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-15T22:12:37.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-04-15T22:12:37.444 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-04-15T22:12:37.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-04-15T22:12:37.509 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-04-15T22:12:37.512 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T22:12:37.512 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-04-15T22:12:37.538 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-04-15T22:12:37.539 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-04-15T22:12:37.568 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-04-15T22:12:37.568 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-04-15T22:12:37.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T22:12:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:12:37.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T22:12:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:12:37.641 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T22:12:37.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-04-15T22:12:37.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-04-15T22:12:37.719 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T22:12:37.729 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T22:12:37.752 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T22:12:37.754 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T22:17:10.671 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2024-04-15T22:17:10.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-15T22:17:10.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:17:10.770 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-15T22:17:10.857 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-15T22:17:10.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:17:10.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:17:10.957 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-04-15T22:17:11.048 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-15T22:17:11.049 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T22:17:11.061 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T22:17:11.061 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T22:17:11.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:17:11.064 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T22:17:11.064 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:17:11.096 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T22:17:11.111 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T22:17:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T22:17:11.142 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-04-15T22:17:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T22:17:11.156 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-04-15T22:17:11.174 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-04-15T22:17:11.174 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T22:17:11.188 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-04-15T22:17:11.191 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T22:17:11.206 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T22:17:11.223 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T22:17:11.251 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:17:11.254 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:17:11.254 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-04-15T22:17:11.254 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.254 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.254 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.254 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.276 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:17:11.279 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:17:11.279 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-04-15T22:17:11.279 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.279 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.279 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.279 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:17:11.280 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-15T22:17:11.406 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'} 2024-04-15T22:17:11.406 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-15T22:17:11.406 INFO:tasks.cephadm:Cluster fsid is e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:11.407 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-15T22:17:11.407 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-15T22:17:11.407 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi081': '172.21.15.81', 'mon.smithi163': '172.21.15.163'} 2024-04-15T22:17:11.407 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-15T22:17:11.407 DEBUG:teuthology.orchestra.run.smithi081:> sudo hostname $(hostname -s) 2024-04-15T22:17:11.437 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-04-15T22:17:11.467 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-15T22:17:11.468 DEBUG:teuthology.orchestra.run.smithi081:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T22:17:11.666 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 15 22:17 /home/ubuntu/cephtest/cephadm 2024-04-15T22:17:11.666 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T22:17:11.856 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 15 22:17 /home/ubuntu/cephtest/cephadm 2024-04-15T22:17:11.856 DEBUG:teuthology.orchestra.run.smithi081:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T22:17:11.873 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T22:17:11.898 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-15T22:17:11.899 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-15T22:17:11.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-15T22:17:12.761 INFO:teuthology.orchestra.run.smithi081.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:17:12.802 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:17:35.996 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-15T22:17:35.997 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-15T22:17:35.997 INFO:teuthology.orchestra.run.smithi081.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-15T22:17:35.997 INFO:teuthology.orchestra.run.smithi081.stdout: "repo_digests": [ 2024-04-15T22:17:35.997 INFO:teuthology.orchestra.run.smithi081.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-15T22:17:35.997 INFO:teuthology.orchestra.run.smithi081.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-15T22:17:35.997 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-15T22:17:35.997 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-15T22:17:38.146 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-04-15T22:17:38.146 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-15T22:17:38.146 INFO:teuthology.orchestra.run.smithi163.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-15T22:17:38.146 INFO:teuthology.orchestra.run.smithi163.stdout: "repo_digests": [ 2024-04-15T22:17:38.147 INFO:teuthology.orchestra.run.smithi163.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-15T22:17:38.147 INFO:teuthology.orchestra.run.smithi163.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-15T22:17:38.147 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-04-15T22:17:38.147 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-04-15T22:17:38.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph 2024-04-15T22:17:38.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-04-15T22:17:38.234 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /etc/ceph 2024-04-15T22:17:38.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-04-15T22:17:38.302 INFO:tasks.cephadm:Writing seed config... 2024-04-15T22:17:38.303 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-15T22:17:38.304 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-15T22:17:38.304 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-15T22:17:38.304 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-15T22:17:38.304 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-15T22:17:38.304 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-15T22:17:38.304 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-15T22:17:38.304 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-15T22:17:38.305 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:17:38.305 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-15T22:17:38.330 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e7d16278-fb75-11ee-bc8f-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-15T22:17:38.334 DEBUG:teuthology.orchestra.run.smithi081:mon.smithi081> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service 2024-04-15T22:17:38.372 INFO:tasks.cephadm:Bootstrapping... 2024-04-15T22:17:38.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.81 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:17:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:-------------------------------------------------------------------------------- 2024-04-15T22:17:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'e7d16278-fb75-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.81', '--skip-admin-label'] 2024-04-15T22:17:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: 4.9.4-dev 2024-04-15T22:17:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:Verifying podman|docker is present... 2024-04-15T22:17:38.614 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: 4.9.4-dev 2024-04-15T22:17:38.617 INFO:teuthology.orchestra.run.smithi081.stderr:Verifying lvm2 is present... 2024-04-15T22:17:38.617 INFO:teuthology.orchestra.run.smithi081.stderr:Verifying time synchronization is in place... 2024-04-15T22:17:38.624 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T22:17:38.631 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: inactive 2024-04-15T22:17:38.638 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: enabled 2024-04-15T22:17:38.644 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: active 2024-04-15T22:17:38.645 INFO:teuthology.orchestra.run.smithi081.stderr:Unit chronyd.service is enabled and running 2024-04-15T22:17:38.645 INFO:teuthology.orchestra.run.smithi081.stderr:Repeating the final host check... 2024-04-15T22:17:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: 4.9.4-dev 2024-04-15T22:17:38.669 INFO:teuthology.orchestra.run.smithi081.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-15T22:17:38.669 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl is present 2024-04-15T22:17:38.669 INFO:teuthology.orchestra.run.smithi081.stderr:lvcreate is present 2024-04-15T22:17:38.675 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T22:17:38.681 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: inactive 2024-04-15T22:17:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: enabled 2024-04-15T22:17:38.694 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: active 2024-04-15T22:17:38.695 INFO:teuthology.orchestra.run.smithi081.stderr:Unit chronyd.service is enabled and running 2024-04-15T22:17:38.695 INFO:teuthology.orchestra.run.smithi081.stderr:Host looks OK 2024-04-15T22:17:38.696 INFO:teuthology.orchestra.run.smithi081.stderr:Cluster fsid: e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:38.696 INFO:teuthology.orchestra.run.smithi081.stderr:Acquiring lock 139737609870736 on /run/cephadm/e7d16278-fb75-11ee-bc8f-c7b262605968.lock 2024-04-15T22:17:38.696 INFO:teuthology.orchestra.run.smithi081.stderr:Lock 139737609870736 acquired on /run/cephadm/e7d16278-fb75-11ee-bc8f-c7b262605968.lock 2024-04-15T22:17:38.696 INFO:teuthology.orchestra.run.smithi081.stderr:Verifying IP 172.21.15.81 port 3300 ... 2024-04-15T22:17:38.696 INFO:teuthology.orchestra.run.smithi081.stderr:Verifying IP 172.21.15.81 port 6789 ... 2024-04-15T22:17:38.696 INFO:teuthology.orchestra.run.smithi081.stderr:Base mon IP is 172.21.15.81, final addrv is [v2:172.21.15.81:3300,v1:172.21.15.81:6789] 2024-04-15T22:17:38.699 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.81 metric 100 2024-04-15T22:17:38.699 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.81 metric 100 2024-04-15T22:17:38.702 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-15T22:17:38.702 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-15T22:17:38.702 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1783sec hoplimit 64 pref medium 2024-04-15T22:17:38.704 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-15T22:17:38.704 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-15T22:17:38.705 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-15T22:17:38.705 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-15T22:17:38.705 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-04-15T22:17:38.705 INFO:teuthology.orchestra.run.smithi081.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-15T22:17:38.705 INFO:teuthology.orchestra.run.smithi081.stderr:Mon IP `172.21.15.81` is in CIDR network `172.21.0.0/20` 2024-04-15T22:17:38.705 INFO:teuthology.orchestra.run.smithi081.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-15T22:17:38.706 INFO:teuthology.orchestra.run.smithi081.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:17:38.810 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:17:39.325 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Getting image source signatures 2024-04-15T22:17:39.325 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-15T22:17:39.325 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-15T22:17:39.325 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-15T22:17:39.325 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-15T22:17:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-15T22:17:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-15T22:17:39.491 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: Writing manifest to image destination 2024-04-15T22:17:39.644 INFO:teuthology.orchestra.run.smithi081.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-15T22:17:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-15T22:17:40.744 INFO:teuthology.orchestra.run.smithi081.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-15T22:17:40.744 INFO:teuthology.orchestra.run.smithi081.stderr:Extracting ceph user uid/gid from container image... 2024-04-15T22:17:41.151 INFO:teuthology.orchestra.run.smithi081.stderr:stat: 167 167 2024-04-15T22:17:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:Creating initial keys... 2024-04-15T22:17:42.090 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph-authtool: AQCGpx1mefRVABAAn0SH3K/isSBI9sn3u6MweQ== 2024-04-15T22:17:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph-authtool: AQCGpx1m2KljNhAAWz4mus1OWWvirNqGOMoAkQ== 2024-04-15T22:17:43.884 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph-authtool: AQCHpx1mph02LxAAAeO6aWRpSG+12JJWZUbiiA== 2024-04-15T22:17:44.425 INFO:teuthology.orchestra.run.smithi081.stderr:Creating initial monmap... 2024-04-15T22:17:44.790 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T22:17:44.790 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-15T22:17:44.791 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:44.791 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T22:17:45.271 INFO:teuthology.orchestra.run.smithi081.stderr:monmaptool for smithi081 [v2:172.21.15.81:3300,v1:172.21.15.81:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T22:17:45.271 INFO:teuthology.orchestra.run.smithi081.stderr:setting min_mon_release = octopus 2024-04-15T22:17:45.271 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/monmaptool: set fsid to e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:45.271 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T22:17:45.272 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:17:45.272 INFO:teuthology.orchestra.run.smithi081.stderr:Creating mon... 2024-04-15T22:17:46.211 INFO:teuthology.orchestra.run.smithi081.stderr:create mon.smithi081 on 2024-04-15T22:17:46.404 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-15T22:17:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968.target. 2024-04-15T22:17:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968.target. 2024-04-15T22:17:46.985 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Failed to reset failed state of unit ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service: Unit ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service not loaded. 2024-04-15T22:17:46.993 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968.target.wants/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service → /etc/systemd/system/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@.service. 2024-04-15T22:17:47.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:47 smithi081 podman[24482]: 2024-04-15 22:17:47.411667252 +0000 UTC m=+0.018151745 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-15T22:17:48.558 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:48 smithi081 podman[24482]: 2024-04-15 22:17:48.300247548 +0000 UTC m=+0.906732037 container create db103c4d8f80eabe933b9e346c0fbf64136b58403bd01a6eb56f2d441e37a3f1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_CLEAN=True, name=centos-stream, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.tags=base centos centos-stream) 2024-04-15T22:17:50.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:50 smithi081 podman[24482]: 2024-04-15 22:17:50.225174475 +0000 UTC m=+2.831658965 container init db103c4d8f80eabe933b9e346c0fbf64136b58403bd01a6eb56f2d441e37a3f1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, architecture=x86_64, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, version=8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream) 2024-04-15T22:17:50.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:50 smithi081 podman[24482]: 2024-04-15 22:17:50.22878231 +0000 UTC m=+2.835266801 container start db103c4d8f80eabe933b9e346c0fbf64136b58403bd01a6eb56f2d441e37a3f1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, architecture=x86_64, vcs-type=git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , name=centos-stream, release=754, GIT_CLEAN=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.expose-services=) 2024-04-15T22:17:50.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:50 smithi081 ceph-mon[24500]: mkfs e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:50.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:50 smithi081 ceph-mon[24500]: mon.smithi081 is new leader, mons smithi081 in quorum (ranks 0) 2024-04-15T22:17:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: disabled 2024-04-15T22:17:51.330 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: inactive 2024-04-15T22:17:51.331 INFO:teuthology.orchestra.run.smithi081.stderr:firewalld.service is not enabled 2024-04-15T22:17:51.331 INFO:teuthology.orchestra.run.smithi081.stderr:Not possible to enable service . firewalld.service is not available 2024-04-15T22:17:51.332 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for mon to start... 2024-04-15T22:17:51.332 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for mon... 2024-04-15T22:17:51.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:51 smithi081 ceph-mon[24500]: mon.smithi081 is new leader, mons smithi081 in quorum (ranks 0) 2024-04-15T22:17:51.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:51 smithi081 ceph-mon[24500]: monmap e1: 1 mons at {smithi081=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0]} 2024-04-15T22:17:51.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:51 smithi081 ceph-mon[24500]: fsmap 2024-04-15T22:17:51.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:51 smithi081 ceph-mon[24500]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T22:17:51.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:51 smithi081 ceph-mon[24500]: mgrmap e1: no daemons active 2024-04-15T22:17:51.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:51 smithi081 bash[24482]: db103c4d8f80eabe933b9e346c0fbf64136b58403bd01a6eb56f2d441e37a3f1 2024-04-15T22:17:51.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:51 smithi081 systemd[1]: Started Ceph mon.smithi081 for e7d16278-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:17:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:17:54 smithi081 ceph-mon[24500]: from='client.? 172.21.15.81:0/1120143163' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-15T22:17:55.329 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: cluster: 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: id: e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: services: 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi081 (age 4s) 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: data: 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: pgs: 2024-04-15T22:17:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:17:59.785 INFO:teuthology.orchestra.run.smithi081.stderr:mon is available 2024-04-15T22:17:59.786 INFO:teuthology.orchestra.run.smithi081.stderr:Assimilating anything we can from ceph.conf... 2024-04-15T22:18:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: [global] 2024-04-15T22:18:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: fsid = e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.81:3300,v1:172.21.15.81:6789] 2024-04-15T22:18:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-15T22:18:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: [mgr] 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: [osd] 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-15T22:18:00.294 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-15T22:18:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:Generating new minimal ceph.conf... 2024-04-15T22:18:01.379 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:01 smithi081 ceph-mon[24500]: from='client.? 172.21.15.81:0/3294061778' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T22:18:01.379 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:01 smithi081 ceph-mon[24500]: from='client.? 172.21.15.81:0/3294061778' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-15T22:18:01.924 INFO:teuthology.orchestra.run.smithi081.stderr:Restarting the monitor... 2024-04-15T22:18:02.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:01 smithi081 systemd[1]: Stopping Ceph mon.smithi081 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:18:02.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:01 smithi081 bash[24732]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon.smithi081" found: no such container 2024-04-15T22:18:02.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081[24496]: 2024-04-15T22:18:02.091+0000 7f3637fd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi081 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:18:02.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081[24496]: 2024-04-15T22:18:02.091+0000 7f3637fd3700 -1 mon.smithi081@0(leader) e1 *** Got Signal Terminated *** 2024-04-15T22:18:02.607 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 podman[24745]: 2024-04-15 22:18:02.326893893 +0000 UTC m=+0.344613072 container died db103c4d8f80eabe933b9e346c0fbf64136b58403bd01a6eb56f2d441e37a3f1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, name=centos-stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T22:18:02.949 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 podman[24745]: 2024-04-15 22:18:02.707035488 +0000 UTC m=+0.724754668 container cleanup db103c4d8f80eabe933b9e346c0fbf64136b58403bd01a6eb56f2d441e37a3f1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-type=git, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, name=centos-stream, io.buildah.version=1.19.8) 2024-04-15T22:18:02.950 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 bash[24745]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081 2024-04-15T22:18:02.950 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 bash[24772]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon.smithi081" found: no such container 2024-04-15T22:18:03.276 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 podman[24758]: 2024-04-15 22:18:02.949551645 +0000 UTC m=+0.619627233 container remove db103c4d8f80eabe933b9e346c0fbf64136b58403bd01a6eb56f2d441e37a3f1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, vcs-type=git, name=centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, distribution-scope=public, architecture=x86_64, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8) 2024-04-15T22:18:03.276 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 systemd[1]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service: Deactivated successfully. 2024-04-15T22:18:03.276 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 systemd[1]: Stopped Ceph mon.smithi081 for e7d16278-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:18:03.276 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:02 smithi081 systemd[1]: Starting Ceph mon.smithi081 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:18:03.276 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 podman[24841]: 2024-04-15 22:18:03.17544238 +0000 UTC m=+0.111496367 container create 8622084b079c62c9d2612202ad210ceb1f7f951fa56bc884d6e2514f4c0387ae (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, ceph=True, maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T22:18:03.276 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 podman[24841]: 2024-04-15 22:18:03.082041473 +0000 UTC m=+0.018095409 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-15T22:18:03.464 INFO:teuthology.orchestra.run.smithi081.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-15T22:18:03.697 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 podman[24841]: 2024-04-15 22:18:03.377469722 +0000 UTC m=+0.313523646 container init 8622084b079c62c9d2612202ad210ceb1f7f951fa56bc884d6e2514f4c0387ae (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., name=centos-stream, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, RELEASE=HEAD, version=8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public) 2024-04-15T22:18:03.698 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 podman[24841]: 2024-04-15 22:18:03.380595855 +0000 UTC m=+0.316649780 container start 8622084b079c62c9d2612202ad210ceb1f7f951fa56bc884d6e2514f4c0387ae (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, GIT_CLEAN=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, RELEASE=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8) 2024-04-15T22:18:03.698 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T22:18:03.698 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-15T22:18:03.698 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: pidfile_write: ignore empty --pid-file 2024-04-15T22:18:03.698 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: load: jerasure load: lrc 2024-04-15T22:18:03.698 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: RocksDB version: 6.15.5 2024-04-15T22:18:03.699 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-15T22:18:03.699 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Compile date Apr 18 2022 2024-04-15T22:18:03.699 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: DB SUMMARY 2024-04-15T22:18:03.699 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: DB Session ID: AEWV3PVABUQS0KUP9Q2J 2024-04-15T22:18:03.699 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: CURRENT file: CURRENT 2024-04-15T22:18:03.699 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T22:18:03.699 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-15T22:18:03.700 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi081/store.db dir, Total Num: 1, files: 000008.sst 2024-04-15T22:18:03.700 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi081/store.db: 000010.log size: 82613 ; 2024-04-15T22:18:03.700 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.error_if_exists: 0 2024-04-15T22:18:03.700 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.create_if_missing: 0 2024-04-15T22:18:03.700 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.paranoid_checks: 1 2024-04-15T22:18:03.700 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T22:18:03.700 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.env: 0x555f42ec6860 2024-04-15T22:18:03.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.fs: Posix File System 2024-04-15T22:18:03.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.info_log: 0x555f442f5fa0 2024-04-15T22:18:03.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T22:18:03.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.statistics: (nil) 2024-04-15T22:18:03.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.use_fsync: 0 2024-04-15T22:18:03.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_log_file_size: 0 2024-04-15T22:18:03.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T22:18:03.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T22:18:03.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T22:18:03.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T22:18:03.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.allow_fallocate: 1 2024-04-15T22:18:03.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.use_direct_reads: 0 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.db_log_dir: 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi081/store.db 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T22:18:03.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T22:18:03.704 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T22:18:03.704 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T22:18:03.704 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T22:18:03.706 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T22:18:03.706 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.write_buffer_manager: 0x555f443e6330 2024-04-15T22:18:03.706 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T22:18:03.706 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-15T22:18:03.706 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T22:18:03.707 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T22:18:03.707 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T22:18:03.707 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T22:18:03.707 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T22:18:03.707 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T22:18:03.707 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T22:18:03.708 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.unordered_write: 0 2024-04-15T22:18:03.708 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T22:18:03.708 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T22:18:03.708 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T22:18:03.708 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T22:18:03.708 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.row_cache: None 2024-04-15T22:18:03.708 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.wal_filter: None 2024-04-15T22:18:03.709 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T22:18:03.709 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T22:18:03.709 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.preserve_deletes: 0 2024-04-15T22:18:03.709 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.two_write_queues: 0 2024-04-15T22:18:03.709 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T22:18:03.709 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.atomic_flush: 0 2024-04-15T22:18:03.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T22:18:03.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T22:18:03.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T22:18:03.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.log_readahead_size: 0 2024-04-15T22:18:03.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T22:18:03.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T22:18:03.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T22:18:03.711 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T22:18:03.711 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T22:18:03.711 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T22:18:03.711 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_background_jobs: 2 2024-04-15T22:18:03.711 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_background_compactions: -1 2024-04-15T22:18:03.711 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_subcompactions: 1 2024-04-15T22:18:03.711 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T22:18:03.712 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T22:18:03.712 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T22:18:03.712 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T22:18:03.712 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T22:18:03.712 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T22:18:03.712 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T22:18:03.712 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T22:18:03.713 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_open_files: -1 2024-04-15T22:18:03.713 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T22:18:03.713 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T22:18:03.713 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T22:18:03.713 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T22:18:03.713 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_background_flushes: -1 2024-04-15T22:18:03.714 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Compression algorithms supported: 2024-04-15T22:18:03.714 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T22:18:03.714 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kZSTD supported: 0 2024-04-15T22:18:03.714 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kXpressCompression supported: 0 2024-04-15T22:18:03.714 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T22:18:03.714 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kLZ4Compression supported: 1 2024-04-15T22:18:03.715 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kBZip2Compression supported: 0 2024-04-15T22:18:03.715 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kZlibCompression supported: 1 2024-04-15T22:18:03.715 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: kSnappyCompression supported: 1 2024-04-15T22:18:03.715 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T22:18:03.715 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi081/store.db/MANIFEST-000009 2024-04-15T22:18:03.715 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-15T22:18:03.716 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T22:18:03.716 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.merge_operator: 2024-04-15T22:18:03.716 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_filter: None 2024-04-15T22:18:03.716 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T22:18:03.716 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T22:18:03.716 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T22:18:03.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T22:18:03.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555f442c1d00) 2024-04-15T22:18:03.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: cache_index_and_filter_blocks: 1 2024-04-15T22:18:03.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T22:18:03.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T22:18:03.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: pin_top_level_index_and_filter: 1 2024-04-15T22:18:03.718 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: index_type: 0 2024-04-15T22:18:03.718 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: data_block_index_type: 0 2024-04-15T22:18:03.718 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: index_shortening: 1 2024-04-15T22:18:03.718 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T22:18:03.718 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: hash_index_allow_collision: 1 2024-04-15T22:18:03.719 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: checksum: 1 2024-04-15T22:18:03.719 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: no_block_cache: 0 2024-04-15T22:18:03.719 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_cache: 0x555f4432c170 2024-04-15T22:18:03.719 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_cache_name: BinnedLRUCache 2024-04-15T22:18:03.719 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_cache_options: 2024-04-15T22:18:03.719 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: capacity : 536870912 2024-04-15T22:18:03.720 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: num_shard_bits : 4 2024-04-15T22:18:03.720 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: strict_capacity_limit : 0 2024-04-15T22:18:03.720 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: high_pri_pool_ratio: 0.000 2024-04-15T22:18:03.720 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_cache_compressed: (nil) 2024-04-15T22:18:03.720 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: persistent_cache: (nil) 2024-04-15T22:18:03.720 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_size: 4096 2024-04-15T22:18:03.721 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_size_deviation: 10 2024-04-15T22:18:03.721 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_restart_interval: 16 2024-04-15T22:18:03.721 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: index_block_restart_interval: 1 2024-04-15T22:18:03.721 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: metadata_block_size: 4096 2024-04-15T22:18:03.721 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: partition_filters: 0 2024-04-15T22:18:03.721 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: use_delta_encoding: 1 2024-04-15T22:18:03.721 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-15T22:18:03.722 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: whole_key_filtering: 1 2024-04-15T22:18:03.722 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: verify_compression: 0 2024-04-15T22:18:03.722 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: read_amp_bytes_per_bit: 0 2024-04-15T22:18:03.722 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: format_version: 4 2024-04-15T22:18:03.722 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: enable_index_compression: 1 2024-04-15T22:18:03.722 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: block_align: 0 2024-04-15T22:18:03.722 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T22:18:03.723 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T22:18:03.723 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression: NoCompression 2024-04-15T22:18:03.723 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T22:18:03.723 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T22:18:03.723 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T22:18:03.723 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.num_levels: 7 2024-04-15T22:18:03.724 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T22:18:03.724 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T22:18:03.724 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T22:18:03.724 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T22:18:03.724 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T22:18:03.724 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T22:18:03.724 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T22:18:03.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:18:03.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T22:18:03.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T22:18:03.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T22:18:03.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T22:18:03.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T22:18:03.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T22:18:03.726 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:18:03.726 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T22:18:03.726 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T22:18:03.726 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T22:18:03.726 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T22:18:03.726 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T22:18:03.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.arena_block_size: 4194304 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T22:18:03.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T22:18:03.729 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T22:18:03.730 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T22:18:03.730 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T22:18:03.730 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T22:18:03.730 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T22:18:03.730 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.table_properties_collectors: 2024-04-15T22:18:03.730 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.inplace_update_support: 0 2024-04-15T22:18:03.730 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T22:18:03.731 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T22:18:03.731 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T22:18:03.731 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T22:18:03.731 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.bloom_locality: 0 2024-04-15T22:18:03.731 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.max_successive_merges: 0 2024-04-15T22:18:03.731 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T22:18:03.731 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T22:18:03.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T22:18:03.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T22:18:03.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.ttl: 2592000 2024-04-15T22:18:03.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T22:18:03.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.enable_blob_files: false 2024-04-15T22:18:03.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.min_blob_size: 0 2024-04-15T22:18:03.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T22:18:03.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T22:18:03.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T22:18:03.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T22:18:03.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi081/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T22:18:03.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-15T22:18:03.734 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-15T22:18:03.734 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219483406248, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-15T22:18:03.734 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-15T22:18:03.734 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-15T22:18:03.734 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219483406942, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 78819, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77082, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9889, "raw_average_key_size": 47, "raw_value_size": 71377, "raw_average_value_size": 339, "num_data_blocks": 10, "num_entries": 210, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713219483, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e369bc67-0fb7-4592-82ed-50d535bbb8fb", "db_session_id": "AEWV3PVABUQS0KUP9Q2J"}} 2024-04-15T22:18:03.734 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-15T22:18:03.735 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219483407262, "job": 1, "event": "recovery_finished"} 2024-04-15T22:18:03.735 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi081/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T22:18:03.735 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x555f44312a80 2024-04-15T22:18:03.735 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: DB pointer 0x555f44322000 2024-04-15T22:18:03.735 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-15T22:18:03.735 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-15T22:18:03.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ** DB Stats ** 2024-04-15T22:18:03.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:18:03.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T22:18:03.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T22:18:03.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:18:03.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T22:18:03.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: L0 2/0 78.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.7 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Sum 2/0 78.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.7 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:03.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.7 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.7 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:03.738 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Cumulative compaction: 0.00 GB write, 25.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Interval compaction: 0.00 GB write, 25.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: 2024-04-15T22:18:03.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: L0 2/0 78.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.7 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Sum 2/0 78.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.7 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:03.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:03.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.7 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:03.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:18:03.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:03.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:03.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:18:03.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:18:03.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:18:03.742 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Cumulative compaction: 0.00 GB write, 25.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:03.742 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:03.742 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:18:03.742 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: 2024-04-15T22:18:03.742 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:18:03.742 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: starting mon.smithi081 rank 0 at public addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] at bind addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi081 fsid e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:03.743 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???) e1 preinit fsid e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:03.743 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???).mds e1 new map 2024-04-15T22:18:03.743 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???).mds e1 print_map 2024-04-15T22:18:03.743 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: e1 2024-04-15T22:18:03.743 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T22:18:03.743 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:18:03.743 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: legacy client fscid: -1 2024-04-15T22:18:03.744 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: 2024-04-15T22:18:03.744 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout: No filesystems configured 2024-04-15T22:18:03.744 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T22:18:03.744 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:18:03.744 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:18:03.744 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:18:03.744 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-15T22:18:03.745 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expand map: {default=false} 2024-04-15T22:18:03.745 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta from 'false' to 'false' 2024-04-15T22:18:03.745 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expanded map: {default=false} 2024-04-15T22:18:03.745 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expand map: {default=info} 2024-04-15T22:18:03.745 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta from 'info' to 'info' 2024-04-15T22:18:03.745 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expanded map: {default=info} 2024-04-15T22:18:03.745 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expand map: {default=daemon} 2024-04-15T22:18:03.746 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-15T22:18:03.746 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expanded map: {default=daemon} 2024-04-15T22:18:03.746 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expand map: {default=debug} 2024-04-15T22:18:03.746 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta from 'debug' to 'debug' 2024-04-15T22:18:03.746 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: expand_channel_meta expanded map: {default=debug} 2024-04-15T22:18:03.746 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 bash[24841]: 8622084b079c62c9d2612202ad210ceb1f7f951fa56bc884d6e2514f4c0387ae 2024-04-15T22:18:03.747 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 systemd[1]: Started Ceph mon.smithi081 for e7d16278-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:18:03.747 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mon.smithi081 is new leader, mons smithi081 in quorum (ranks 0) 2024-04-15T22:18:03.747 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: monmap e1: 1 mons at {smithi081=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0]} 2024-04-15T22:18:03.747 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: fsmap 2024-04-15T22:18:03.747 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T22:18:03.747 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:03 smithi081 ceph-mon[24860]: mgrmap e1: no daemons active 2024-04-15T22:18:04.479 INFO:teuthology.orchestra.run.smithi081.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-15T22:18:04.479 INFO:teuthology.orchestra.run.smithi081.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:18:04.479 INFO:teuthology.orchestra.run.smithi081.stderr:Creating mgr... 2024-04-15T22:18:04.480 INFO:teuthology.orchestra.run.smithi081.stderr:Verifying port 9283 ... 2024-04-15T22:18:04.725 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Failed to reset failed state of unit ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mgr.smithi081.wfvsja.service: Unit ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mgr.smithi081.wfvsja.service not loaded. 2024-04-15T22:18:04.733 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968.target.wants/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mgr.smithi081.wfvsja.service → /etc/systemd/system/ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@.service. 2024-04-15T22:18:05.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:04 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/434508950' entity='client.admin' 2024-04-15T22:18:05.957 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: disabled 2024-04-15T22:18:05.964 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: inactive 2024-04-15T22:18:05.965 INFO:teuthology.orchestra.run.smithi081.stderr:firewalld.service is not enabled 2024-04-15T22:18:05.965 INFO:teuthology.orchestra.run.smithi081.stderr:Not possible to enable service . firewalld.service is not available 2024-04-15T22:18:05.973 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: disabled 2024-04-15T22:18:05.980 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: inactive 2024-04-15T22:18:05.981 INFO:teuthology.orchestra.run.smithi081.stderr:firewalld.service is not enabled 2024-04-15T22:18:05.981 INFO:teuthology.orchestra.run.smithi081.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-15T22:18:05.981 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for mgr to start... 2024-04-15T22:18:05.981 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for mgr... 2024-04-15T22:18:06.522 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:06.522 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:06.522 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsid": "e7d16278-fb75-11ee-bc8f-c7b262605968", 2024-04-15T22:18:06.522 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "health": { 2024-04-15T22:18:06.522 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 0 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "smithi081" 2024-04-15T22:18:06.523 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_age": 3, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:06.524 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T22:18:06.525 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "available": false, 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "iostat", 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "nfs", 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "restful" 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modified": "2024-04-15T22:17:50.268827+0000", 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T22:18:06.526 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:06.807 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:06 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3131339593' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:18:07.069 INFO:teuthology.orchestra.run.smithi081.stderr:mgr not available, waiting (1/15)... 2024-04-15T22:18:09.618 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:09.618 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:09.618 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsid": "e7d16278-fb75-11ee-bc8f-c7b262605968", 2024-04-15T22:18:09.618 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "health": { 2024-04-15T22:18:09.618 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 0 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "smithi081" 2024-04-15T22:18:09.619 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T22:18:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "available": false, 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "iostat", 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "nfs", 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "restful" 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modified": "2024-04-15T22:17:50.268827+0000", 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:09.623 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T22:18:09.623 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:09.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:09 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/674267534' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:18:10.158 INFO:teuthology.orchestra.run.smithi081.stderr:mgr not available, waiting (2/15)... 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: Activating manager daemon smithi081.wfvsja 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: mgrmap e2: smithi081.wfvsja(active, starting, since 0.00269304s) 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr metadata", "who": "smithi081.wfvsja", "id": "smithi081.wfvsja"}]: dispatch 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: Manager daemon smithi081.wfvsja is now available 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:18:12.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:12.057 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:12.057 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:11 smithi081 ceph-mon[24860]: from='mgr.14100 172.21.15.81:0/2007848564' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsid": "e7d16278-fb75-11ee-bc8f-c7b262605968", 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "health": { 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T22:18:12.646 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 0 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "smithi081" 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T22:18:12.647 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "available": false, 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "iostat", 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "nfs", 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "restful" 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:12.649 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modified": "2024-04-15T22:17:50.268827+0000", 2024-04-15T22:18:12.650 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:12.650 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:12.650 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T22:18:12.650 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:12.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:12 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2403083532' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:18:12.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:12 smithi081 ceph-mon[24860]: mgrmap e3: smithi081.wfvsja(active, since 1.00546s) 2024-04-15T22:18:13.189 INFO:teuthology.orchestra.run.smithi081.stderr:mgr not available, waiting (3/15)... 2024-04-15T22:18:15.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:15 smithi081 ceph-mon[24860]: mgrmap e4: smithi081.wfvsja(active, since 2s) 2024-04-15T22:18:15.847 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsid": "e7d16278-fb75-11ee-bc8f-c7b262605968", 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "health": { 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 0 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "smithi081" 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:15.848 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T22:18:15.849 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T22:18:15.850 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "available": true, 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "iostat", 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "nfs", 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "restful" 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ], 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "modified": "2024-04-15T22:17:50.268827+0000", 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:18:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: }, 2024-04-15T22:18:15.852 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T22:18:15.852 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:16.412 INFO:teuthology.orchestra.run.smithi081.stderr:mgr is available 2024-04-15T22:18:16.736 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:16 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2830995956' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: [global] 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: fsid = e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: [mgr] 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 2024-04-15T22:18:17.015 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: [osd] 2024-04-15T22:18:17.016 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-15T22:18:17.016 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-15T22:18:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:Enabling cephadm module... 2024-04-15T22:18:17.674 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:17 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/542877547' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T22:18:18.737 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:18 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1295384850' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-15T22:18:19.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:19 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1295384850' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-15T22:18:19.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:19 smithi081 ceph-mon[24860]: mgrmap e5: smithi081.wfvsja(active, since 6s) 2024-04-15T22:18:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-15T22:18:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "available": true, 2024-04-15T22:18:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "active_name": "smithi081.wfvsja", 2024-04-15T22:18:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-15T22:18:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:20.238 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for the mgr to restart... 2024-04-15T22:18:20.238 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for mgr epoch 5... 2024-04-15T22:18:20.466 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:20 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/497867252' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T22:18:25.425 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:25.426 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-15T22:18:25.426 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "initialized": true 2024-04-15T22:18:25.426 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:25.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: Active manager daemon smithi081.wfvsja restarted 2024-04-15T22:18:25.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: Activating manager daemon smithi081.wfvsja 2024-04-15T22:18:25.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: osdmap e2: 0 total, 0 up, 0 in 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: mgrmap e6: smithi081.wfvsja(active, starting, since 0.188865s) 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr metadata", "who": "smithi081.wfvsja", "id": "smithi081.wfvsja"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: Manager daemon smithi081.wfvsja is now available 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:18:25.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:25 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:18:26.027 INFO:teuthology.orchestra.run.smithi081.stderr:mgr epoch 5 is available 2024-04-15T22:18:26.027 INFO:teuthology.orchestra.run.smithi081.stderr:Setting orchestrator backend to cephadm... 2024-04-15T22:18:26.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:26 smithi081 ceph-mon[24860]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T22:18:26.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:26 smithi081 ceph-mon[24860]: mgrmap e7: smithi081.wfvsja(active, since 1.191s) 2024-04-15T22:18:26.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:26 smithi081 ceph-mon[24860]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T22:18:26.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:26 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:26.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:26 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:27.516 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:27 smithi081 ceph-mon[24860]: [15/Apr/2024:22:18:26] ENGINE Bus STARTING 2024-04-15T22:18:27.517 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:27 smithi081 ceph-mon[24860]: [15/Apr/2024:22:18:26] ENGINE Serving on https://172.21.15.81:7150 2024-04-15T22:18:27.517 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:27 smithi081 ceph-mon[24860]: [15/Apr/2024:22:18:26] ENGINE Bus STARTED 2024-04-15T22:18:27.517 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:27 smithi081 ceph-mon[24860]: mgrmap e8: smithi081.wfvsja(active, since 2s) 2024-04-15T22:18:27.517 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:27 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:27.517 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:27 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:27.698 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: value unchanged 2024-04-15T22:18:28.405 INFO:teuthology.orchestra.run.smithi081.stderr:Generating ssh key... 2024-04-15T22:18:28.570 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:28 smithi081 ceph-mon[24860]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:29.749 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:29 smithi081 ceph-mon[24860]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:30.614 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:30 smithi081 ceph-mon[24860]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:30.615 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:30 smithi081 ceph-mon[24860]: Generating ssh key... 2024-04-15T22:18:30.615 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:30 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:30.615 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:30 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:30.615 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLShxrLyP6/4OC4yfeGqQhX4IMjj41ZBZ49IbAq9wJ4YAZwnxB9BXFpQdaX2mxdImFhW+qvvpWnaXgO8FDCeLMB46yPmFa5dF3xp67m8FOgUYxfsm9OA+7oisSMjvej4s+YyRo01LxXcKS5YFRxXyXa61fLU9EZfXZWb3wswMdVkQlwcDlsceHmxg2VieI02Iahki+hcFGjrXY1bRnY+foG9uhebWMZy0JnITO+pVnWXmAhsrx5/SNYrLHkGdkHyV8OIZddKy6eL5EJjGEukFanb6CI8ZBD4zWmJFl8pDElZuJ81QATuFPS6PYLb7LN4POX6UOkKt7MR1iRTsCYn3FZkYm4+IEzC4v6jIOTGOgNg2XJHF0EmghPYC57SR3Hal6q8vajxINyRUON/rjiN9Sn3yLlLV0NuW7PfYA9+Lm8y3ltPmqHUyUQUoIFfvIDeMqL3O/tbMxc5lU6n6AZd412p8ReZPwGnD1tsb20FKzL16VuMEcseBoTzgvxpFslAs= ceph-e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:31.211 INFO:teuthology.orchestra.run.smithi081.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-15T22:18:31.211 INFO:teuthology.orchestra.run.smithi081.stderr:Adding key to root@localhost authorized_keys... 2024-04-15T22:18:31.211 INFO:teuthology.orchestra.run.smithi081.stderr:Adding host smithi081... 2024-04-15T22:18:31.791 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:31 smithi081 ceph-mon[24860]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:32.771 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:32 smithi081 ceph-mon[24860]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi081", "addr": "172.21.15.81", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:32.771 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:32 smithi081 ceph-mon[24860]: Deploying cephadm binary to smithi081 2024-04-15T22:18:32.780 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Added host 'smithi081' with addr '172.21.15.81' 2024-04-15T22:18:33.358 INFO:teuthology.orchestra.run.smithi081.stderr:Deploying mon service with default placement... 2024-04-15T22:18:34.018 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-15T22:18:34.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:33 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:34.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:33 smithi081 ceph-mon[24860]: Added host smithi081 2024-04-15T22:18:34.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:33 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:34.783 INFO:teuthology.orchestra.run.smithi081.stderr:Deploying mgr service with default placement... 2024-04-15T22:18:35.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:35 smithi081 ceph-mon[24860]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:35.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:35 smithi081 ceph-mon[24860]: Saving service mon spec with placement count:5 2024-04-15T22:18:35.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:35 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:35.427 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-15T22:18:36.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:36 smithi081 ceph-mon[24860]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:36.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:36 smithi081 ceph-mon[24860]: Saving service mgr spec with placement count:2 2024-04-15T22:18:36.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:36 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:36.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:36 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:Deploying crash service with default placement... 2024-04-15T22:18:37.292 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-15T22:18:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:Deploying prometheus service with default placement... 2024-04-15T22:18:38.500 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:38 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:38.501 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:38 smithi081 ceph-mon[24860]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:38.501 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:38 smithi081 ceph-mon[24860]: Saving service crash spec with placement * 2024-04-15T22:18:38.501 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:38 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:38.501 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-15T22:18:39.633 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:39 smithi081 ceph-mon[24860]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:39.633 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:39 smithi081 ceph-mon[24860]: Saving service prometheus spec with placement count:1 2024-04-15T22:18:39.633 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:39 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:39.633 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:39 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:39.636 INFO:teuthology.orchestra.run.smithi081.stderr:Deploying grafana service with default placement... 2024-04-15T22:18:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-15T22:18:40.942 INFO:teuthology.orchestra.run.smithi081.stderr:Deploying node-exporter service with default placement... 2024-04-15T22:18:41.270 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:41 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:41.270 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:41 smithi081 ceph-mon[24860]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:41.270 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:41 smithi081 ceph-mon[24860]: Saving service grafana spec with placement count:1 2024-04-15T22:18:41.270 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:41 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:42.090 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-15T22:18:42.846 INFO:teuthology.orchestra.run.smithi081.stderr:Deploying alertmanager service with default placement... 2024-04-15T22:18:43.180 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:43 smithi081 ceph-mon[24860]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:43.180 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:43 smithi081 ceph-mon[24860]: Saving service node-exporter spec with placement * 2024-04-15T22:18:43.180 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:43 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-15T22:18:44.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:44 smithi081 ceph-mon[24860]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:44.701 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:44 smithi081 ceph-mon[24860]: Saving service alertmanager spec with placement count:1 2024-04-15T22:18:44.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:44 smithi081 ceph-mon[24860]: from='mgr.14120 172.21.15.81:0/10443184' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:44.702 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:44 smithi081 ceph-mon[24860]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:45.919 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:45 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/755129607' entity='client.admin' 2024-04-15T22:18:46.558 INFO:teuthology.orchestra.run.smithi081.stderr:Enabling the dashboard module... 2024-04-15T22:18:47.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:46 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2254569756' entity='client.admin' 2024-04-15T22:18:47.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:46 smithi081 ceph-mon[24860]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:48.303 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:47 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1360005012' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:48 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1360005012' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:48 smithi081 ceph-mon[24860]: mgrmap e9: smithi081.wfvsja(active, since 23s) 2024-04-15T22:18:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-15T22:18:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "available": true, 2024-04-15T22:18:49.318 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "active_name": "smithi081.wfvsja", 2024-04-15T22:18:49.318 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-15T22:18:49.318 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:49.905 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for the mgr to restart... 2024-04-15T22:18:49.905 INFO:teuthology.orchestra.run.smithi081.stderr:Waiting for mgr epoch 9... 2024-04-15T22:18:50.126 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:49 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/473812149' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T22:18:54.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:54 smithi081 ceph-mon[24860]: Active manager daemon smithi081.wfvsja restarted 2024-04-15T22:18:54.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:54 smithi081 ceph-mon[24860]: Activating manager daemon smithi081.wfvsja 2024-04-15T22:18:54.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:54 smithi081 ceph-mon[24860]: osdmap e3: 0 total, 0 up, 0 in 2024-04-15T22:18:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: { 2024-04-15T22:18:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-15T22:18:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: "initialized": true 2024-04-15T22:18:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: } 2024-04-15T22:18:55.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: mgrmap e10: smithi081.wfvsja(active, starting, since 0.625173s) 2024-04-15T22:18:55.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:18:55.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr metadata", "who": "smithi081.wfvsja", "id": "smithi081.wfvsja"}]: dispatch 2024-04-15T22:18:55.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:18:55.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:18:55.739 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: Manager daemon smithi081.wfvsja is now available 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: [15/Apr/2024:22:18:54] ENGINE Bus STARTING 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: [15/Apr/2024:22:18:54] ENGINE Serving on https://172.21.15.81:7150 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: [15/Apr/2024:22:18:54] ENGINE Bus STARTED 2024-04-15T22:18:55.740 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:55 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:56.164 INFO:teuthology.orchestra.run.smithi081.stderr:mgr epoch 9 is available 2024-04-15T22:18:56.165 INFO:teuthology.orchestra.run.smithi081.stderr:Generating a dashboard self-signed certificate... 2024-04-15T22:18:56.791 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:56 smithi081 ceph-mon[24860]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T22:18:56.792 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:56 smithi081 ceph-mon[24860]: mgrmap e11: smithi081.wfvsja(active, since 1.62781s) 2024-04-15T22:18:56.792 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:56 smithi081 ceph-mon[24860]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T22:18:56.792 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:56 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:56.995 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-15T22:18:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:Creating initial admin user... 2024-04-15T22:18:57.881 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:57 smithi081 ceph-mon[24860]: mgrmap e12: smithi081.wfvsja(active, since 2s) 2024-04-15T22:18:57.881 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:57 smithi081 ceph-mon[24860]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:57.881 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:57 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:57.881 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:57 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:57.881 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:57 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:18:58.516 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$5AApna7.0GTRvYMo293ewuCZmUaHdVGNorbE7U.lEDrX6FkrY.4/C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713219538, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-15T22:18:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:Fetching dashboard port number... 2024-04-15T22:18:59.787 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:59 smithi081 ceph-mon[24860]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:59.787 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:18:59 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:00.574 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: 8443 2024-04-15T22:19:01.218 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: disabled 2024-04-15T22:19:01.225 INFO:teuthology.orchestra.run.smithi081.stderr:systemctl: inactive 2024-04-15T22:19:01.225 INFO:teuthology.orchestra.run.smithi081.stderr:firewalld.service is not enabled 2024-04-15T22:19:01.226 INFO:teuthology.orchestra.run.smithi081.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-15T22:19:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:Ceph Dashboard is now available at: 2024-04-15T22:19:01.228 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:01.228 INFO:teuthology.orchestra.run.smithi081.stderr: URL: https://smithi081.front.sepia.ceph.com:8443/ 2024-04-15T22:19:01.228 INFO:teuthology.orchestra.run.smithi081.stderr: User: admin 2024-04-15T22:19:01.228 INFO:teuthology.orchestra.run.smithi081.stderr: Password: 5oqeyr3t5r 2024-04-15T22:19:01.229 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:01.229 INFO:teuthology.orchestra.run.smithi081.stderr:Enabling autotune for osd_memory_target 2024-04-15T22:19:01.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:01 smithi081 ceph-mon[24860]: mgrmap e13: smithi081.wfvsja(active, since 6s) 2024-04-15T22:19:01.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:01 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2450615052' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-15T22:19:03.175 INFO:teuthology.orchestra.run.smithi081.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-15T22:19:03.771 INFO:teuthology.orchestra.run.smithi081.stderr:You can access the Ceph CLI with: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: ceph telemetry on 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr:For more information see: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:19:03.772 INFO:teuthology.orchestra.run.smithi081.stderr:Bootstrap complete. 2024-04-15T22:19:03.812 INFO:tasks.cephadm:Fetching config... 2024-04-15T22:19:03.812 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:19:03.813 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-15T22:19:03.830 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-15T22:19:03.830 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:19:03.830 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-15T22:19:03.885 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-15T22:19:03.885 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:19:03.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/keyring of=/dev/stdout 2024-04-15T22:19:03.954 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-15T22:19:03.954 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:19:03.954 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-15T22:19:04.010 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-15T22:19:04.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLShxrLyP6/4OC4yfeGqQhX4IMjj41ZBZ49IbAq9wJ4YAZwnxB9BXFpQdaX2mxdImFhW+qvvpWnaXgO8FDCeLMB46yPmFa5dF3xp67m8FOgUYxfsm9OA+7oisSMjvej4s+YyRo01LxXcKS5YFRxXyXa61fLU9EZfXZWb3wswMdVkQlwcDlsceHmxg2VieI02Iahki+hcFGjrXY1bRnY+foG9uhebWMZy0JnITO+pVnWXmAhsrx5/SNYrLHkGdkHyV8OIZddKy6eL5EJjGEukFanb6CI8ZBD4zWmJFl8pDElZuJ81QATuFPS6PYLb7LN4POX6UOkKt7MR1iRTsCYn3FZkYm4+IEzC4v6jIOTGOgNg2XJHF0EmghPYC57SR3Hal6q8vajxINyRUON/rjiN9Sn3yLlLV0NuW7PfYA9+Lm8y3ltPmqHUyUQUoIFfvIDeMqL3O/tbMxc5lU6n6AZd412p8ReZPwGnD1tsb20FKzL16VuMEcseBoTzgvxpFslAs= ceph-e7d16278-fb75-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T22:19:04.102 INFO:teuthology.orchestra.run.smithi081.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLShxrLyP6/4OC4yfeGqQhX4IMjj41ZBZ49IbAq9wJ4YAZwnxB9BXFpQdaX2mxdImFhW+qvvpWnaXgO8FDCeLMB46yPmFa5dF3xp67m8FOgUYxfsm9OA+7oisSMjvej4s+YyRo01LxXcKS5YFRxXyXa61fLU9EZfXZWb3wswMdVkQlwcDlsceHmxg2VieI02Iahki+hcFGjrXY1bRnY+foG9uhebWMZy0JnITO+pVnWXmAhsrx5/SNYrLHkGdkHyV8OIZddKy6eL5EJjGEukFanb6CI8ZBD4zWmJFl8pDElZuJ81QATuFPS6PYLb7LN4POX6UOkKt7MR1iRTsCYn3FZkYm4+IEzC4v6jIOTGOgNg2XJHF0EmghPYC57SR3Hal6q8vajxINyRUON/rjiN9Sn3yLlLV0NuW7PfYA9+Lm8y3ltPmqHUyUQUoIFfvIDeMqL3O/tbMxc5lU6n6AZd412p8ReZPwGnD1tsb20FKzL16VuMEcseBoTzgvxpFslAs= ceph-e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:19:04.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLShxrLyP6/4OC4yfeGqQhX4IMjj41ZBZ49IbAq9wJ4YAZwnxB9BXFpQdaX2mxdImFhW+qvvpWnaXgO8FDCeLMB46yPmFa5dF3xp67m8FOgUYxfsm9OA+7oisSMjvej4s+YyRo01LxXcKS5YFRxXyXa61fLU9EZfXZWb3wswMdVkQlwcDlsceHmxg2VieI02Iahki+hcFGjrXY1bRnY+foG9uhebWMZy0JnITO+pVnWXmAhsrx5/SNYrLHkGdkHyV8OIZddKy6eL5EJjGEukFanb6CI8ZBD4zWmJFl8pDElZuJ81QATuFPS6PYLb7LN4POX6UOkKt7MR1iRTsCYn3FZkYm4+IEzC4v6jIOTGOgNg2XJHF0EmghPYC57SR3Hal6q8vajxINyRUON/rjiN9Sn3yLlLV0NuW7PfYA9+Lm8y3ltPmqHUyUQUoIFfvIDeMqL3O/tbMxc5lU6n6AZd412p8ReZPwGnD1tsb20FKzL16VuMEcseBoTzgvxpFslAs= ceph-e7d16278-fb75-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T22:19:04.168 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLShxrLyP6/4OC4yfeGqQhX4IMjj41ZBZ49IbAq9wJ4YAZwnxB9BXFpQdaX2mxdImFhW+qvvpWnaXgO8FDCeLMB46yPmFa5dF3xp67m8FOgUYxfsm9OA+7oisSMjvej4s+YyRo01LxXcKS5YFRxXyXa61fLU9EZfXZWb3wswMdVkQlwcDlsceHmxg2VieI02Iahki+hcFGjrXY1bRnY+foG9uhebWMZy0JnITO+pVnWXmAhsrx5/SNYrLHkGdkHyV8OIZddKy6eL5EJjGEukFanb6CI8ZBD4zWmJFl8pDElZuJ81QATuFPS6PYLb7LN4POX6UOkKt7MR1iRTsCYn3FZkYm4+IEzC4v6jIOTGOgNg2XJHF0EmghPYC57SR3Hal6q8vajxINyRUON/rjiN9Sn3yLlLV0NuW7PfYA9+Lm8y3ltPmqHUyUQUoIFfvIDeMqL3O/tbMxc5lU6n6AZd412p8ReZPwGnD1tsb20FKzL16VuMEcseBoTzgvxpFslAs= ceph-e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:19:04.184 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-15T22:19:04.498 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:04 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3904049843' entity='client.admin' 2024-04-15T22:19:07.008 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:06 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:07.008 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:06 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd/host:smithi081", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:19:07.008 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:06 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:07.008 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:06 smithi081 ceph-mon[24860]: Deploying daemon alertmanager.smithi081 on smithi081 2024-04-15T22:19:08.465 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:08 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2385914070' entity='client.admin' 2024-04-15T22:19:08.489 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-15T22:19:08.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-15T22:19:11.795 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi163 2024-04-15T22:19:11.796 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-04-15T22:19:11.796 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.conf 2024-04-15T22:19:11.815 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-04-15T22:19:11.815 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:19:11.874 INFO:tasks.cephadm:Adding host smithi163 to orchestrator... 2024-04-15T22:19:11.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch host add smithi163 2024-04-15T22:19:12.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:11 smithi081 ceph-mon[24860]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:12.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:11 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:15.741 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:15 smithi081 ceph-mon[24860]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:17.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:17 smithi081 ceph-mon[24860]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:17.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:17 smithi081 ceph-mon[24860]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:17.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:17 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:17.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:17 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi081", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:19:17.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:17 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi081", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-15T22:19:17.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:17 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:18.016 INFO:teuthology.orchestra.run.smithi081.stdout:Added host 'smithi163' with addr '172.21.15.163' 2024-04-15T22:19:18.764 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:18 smithi081 ceph-mon[24860]: Deploying cephadm binary to smithi163 2024-04-15T22:19:18.764 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:18 smithi081 ceph-mon[24860]: Deploying daemon crash.smithi081 on smithi081 2024-04-15T22:19:18.764 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:18 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:19.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-15T22:19:19.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:19 smithi081 ceph-mon[24860]: Added host smithi163 2024-04-15T22:19:19.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:19 smithi081 ceph-mon[24860]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:19.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:19 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:19:21.097 INFO:teuthology.orchestra.run.smithi081.stdout:[{"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}] 2024-04-15T22:19:21.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:21 smithi081 ceph-mon[24860]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:21.982 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-15T22:19:21.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-15T22:19:23.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:19:23.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:23.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:23.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:23.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:19:23.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:19:23.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:23.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:22 smithi081 ceph-mon[24860]: Deploying daemon grafana.smithi081 on smithi081 2024-04-15T22:19:24.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:23 smithi081 ceph-mon[24860]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:24.801 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2024-04-15T22:19:25.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:24 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/838636573' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-15T22:19:25.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:24 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:25.415 INFO:tasks.cephadm:Adding mon.smithi081 on smithi081 2024-04-15T22:19:25.415 INFO:tasks.cephadm:Adding mon.smithi163 on smithi163 2024-04-15T22:19:25.415 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch apply mon '2;smithi081:172.21.15.81=smithi081;smithi163:172.21.15.163=smithi163' 2024-04-15T22:19:26.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:25 smithi081 ceph-mon[24860]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:26.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:25 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/838636573' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-15T22:19:26.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:25 smithi081 ceph-mon[24860]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:19:27.243 INFO:teuthology.orchestra.run.smithi163.stdout:Scheduled mon update... 2024-04-15T22:19:27.832 DEBUG:teuthology.orchestra.run.smithi163:mon.smithi163> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi163.service 2024-04-15T22:19:27.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:27.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:28.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:27 smithi081 ceph-mon[24860]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:28.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:27 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:29.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:28 smithi081 ceph-mon[24860]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi081:172.21.15.81=smithi081;smithi163:172.21.15.163=smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:29.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:28 smithi081 ceph-mon[24860]: Saving service mon spec with placement smithi081:172.21.15.81=smithi081;smithi163:172.21.15.163=smithi163;count:2 2024-04-15T22:19:29.642 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:29.642 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:29.644 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:30.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:29 smithi081 ceph-mon[24860]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:30.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:29 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/1120322650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:31.212 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:31.213 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:32.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:31 smithi081 ceph-mon[24860]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:32.987 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:32.989 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:34.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:33 smithi081 ceph-mon[24860]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:34.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:33 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/3716200655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:34.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:34.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:36.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:35 smithi081 ceph-mon[24860]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:36.492 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:36.492 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:36.494 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:36.991 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:36 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/738621242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:37.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:37.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:38.236 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:37 smithi081 ceph-mon[24860]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:38.236 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:37 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:38.236 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:37 smithi081 ceph-mon[24860]: Deploying daemon node-exporter.smithi081 on smithi081 2024-04-15T22:19:39.728 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:39 smithi081 ceph-mon[24860]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:39.788 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:39.788 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:39.790 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:40.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:40 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:40.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:40 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/2954848553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:40.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:40 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:40.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:40 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:19:41.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:41.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:42.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:41 smithi081 ceph-mon[24860]: from='mgr.14162 172.21.15.81:0/1899602270' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:19:42.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:41 smithi081 ceph-mon[24860]: mgrmap e14: smithi081.wfvsja(active, since 47s) 2024-04-15T22:19:43.109 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:43.109 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:43.111 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:43.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:43 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/2332602703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:44.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:44.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:46.588 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:46.589 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:46.591 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:46.705 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:46 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/842196738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:47.972 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:47 smithi081 ceph-mon[24860]: Active manager daemon smithi081.wfvsja restarted 2024-04-15T22:19:47.972 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:47 smithi081 ceph-mon[24860]: Activating manager daemon smithi081.wfvsja 2024-04-15T22:19:47.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:47 smithi081 ceph-mon[24860]: osdmap e5: 0 total, 0 up, 0 in 2024-04-15T22:19:48.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:48.143 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: mgrmap e15: smithi081.wfvsja(active, starting, since 0.937471s) 2024-04-15T22:19:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:19:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr metadata", "who": "smithi081.wfvsja", "id": "smithi081.wfvsja"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: Manager daemon smithi081.wfvsja is now available 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:49.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:49 smithi081 ceph-mon[24860]: [15/Apr/2024:22:19:48] ENGINE Bus STARTING 2024-04-15T22:19:49.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:49 smithi081 ceph-mon[24860]: [15/Apr/2024:22:19:48] ENGINE Serving on https://172.21.15.81:7150 2024-04-15T22:19:49.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:49 smithi081 ceph-mon[24860]: [15/Apr/2024:22:19:48] ENGINE Bus STARTED 2024-04-15T22:19:49.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:49 smithi081 ceph-mon[24860]: mgrmap e16: smithi081.wfvsja(active, since 1.94442s) 2024-04-15T22:19:49.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:49 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:50.034 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:50.034 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:50.036 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:50.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:50 smithi081 ceph-mon[24860]: mgrmap e17: smithi081.wfvsja(active, since 2s) 2024-04-15T22:19:50.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:50.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:50 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/2687270924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:51.613 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:51.613 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:52.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:52 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:52.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:52 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:52.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:52 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd/host:smithi081", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:19:52.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:52 smithi081 ceph-mon[24860]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-15T22:19:53.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:53 smithi081 ceph-mon[24860]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:19:53.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:53 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:53.652 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:53.652 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:53.654 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/481389014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:54.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:54.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:19:54.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:54.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:19:54.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-15T22:19:54.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:55.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:55.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:55.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:55 smithi081 ceph-mon[24860]: Deploying daemon crash.smithi163 on smithi163 2024-04-15T22:19:57.694 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:19:57.694 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:19:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:19:58.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:57 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/2161532245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:59.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:19:59.236 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:19:59.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:19:59.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.cdbjip", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:19:59.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.cdbjip", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T22:19:59.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:19:59.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:59.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:19:59 smithi081 ceph-mon[24860]: Deploying daemon mgr.smithi163.cdbjip on smithi163 2024-04-15T22:20:01.671 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:20:01.671 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:20:01.673 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:20:02.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:01 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/3915591185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:20:03.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:02 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:03.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:02 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:20:03.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:02 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:03.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:02 smithi081 ceph-mon[24860]: Deploying daemon mon.smithi163 on smithi163 2024-04-15T22:20:03.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:02 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:03.528 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:20:03.528 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:20:06.588 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:20:06.588 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:44.712148Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:20:06.590 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-04-15T22:20:07.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:06 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/634311301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:20:08.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:20:08.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:20:12.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: mon.smithi081 calling monitor election 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.? 172.21.15.163:0/2391808567' entity='mgr.smithi163.cdbjip' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi163.cdbjip/crt"}]: dispatch 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: mon.smithi163 calling monitor election 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:12.556 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: mon.smithi081 is new leader, mons smithi081,smithi163 in quorum (ranks 0,1) 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: monmap e2: 2 mons at {smithi081=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],smithi163=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0]} 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: fsmap 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: osdmap e5: 0 total, 0 up, 0 in 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: mgrmap e17: smithi081.wfvsja(active, since 25s) 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: Standby manager daemon smithi163.cdbjip started 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.? 172.21.15.163:0/2391808567' entity='mgr.smithi163.cdbjip' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: overall HEALTH_OK 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.? 172.21.15.163:0/2391808567' entity='mgr.smithi163.cdbjip' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi163.cdbjip/key"}]: dispatch 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.? 172.21.15.163:0/2391808567' entity='mgr.smithi163.cdbjip' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:20:12.557 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:13.409 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-15T22:20:13.409 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":2,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:20:07.003741Z","created":"2024-04-15T22:17:44.712148Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi081","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T22:20:13.410 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 2 2024-04-15T22:20:13.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:13 smithi081 ceph-mon[24860]: Deploying daemon node-exporter.smithi163 on smithi163 2024-04-15T22:20:13.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:13 smithi081 ceph-mon[24860]: mgrmap e18: smithi081.wfvsja(active, since 25s), standbys: smithi163.cdbjip 2024-04-15T22:20:13.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr metadata", "who": "smithi163.cdbjip", "id": "smithi163.cdbjip"}]: dispatch 2024-04-15T22:20:13.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:13.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:20:14.297 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:14 smithi163 ceph-mon[27043]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:14.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:14 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/1040233721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:20:14.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:14 smithi081 ceph-mon[24860]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:14.894 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-15T22:20:14.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-15T22:20:16.799 INFO:teuthology.orchestra.run.smithi081.stdout:# minimal ceph.conf for e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:20:16.799 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-04-15T22:20:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = e7d16278-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:20:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: mon_host = [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-04-15T22:20:17.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:16 smithi081 ceph-mon[24860]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:17.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:16 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:17.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:16 smithi081 ceph-mon[24860]: Deploying daemon prometheus.smithi081 on smithi081 2024-04-15T22:20:17.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:16 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2610430004' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:17.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:16 smithi163 ceph-mon[27043]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:17.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:16 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:17.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:16 smithi163 ceph-mon[27043]: Deploying daemon prometheus.smithi081 on smithi081 2024-04-15T22:20:17.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:16 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2610430004' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:17.378 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-15T22:20:17.378 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:20:17.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T22:20:17.409 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:20:17.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:20:17.479 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-04-15T22:20:17.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T22:20:17.516 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-04-15T22:20:17.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:20:17.589 INFO:tasks.cephadm:Deploying OSDs... 2024-04-15T22:20:17.589 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:20:17.589 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T22:20:17.605 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T22:20:17.605 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-15 22:19:37.588284996 +0000 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-15 22:16:48.229774257 +0000 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-15 22:16:48.229774257 +0000 2024-04-15T22:20:17.661 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-15 22:16:48.229774257 +0000 2024-04-15T22:20:17.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T22:20:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-15T22:20:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-15T22:20:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000254495 s, 2.0 MB/s 2024-04-15T22:20:17.731 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T22:20:17.787 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-15 22:19:37.588284996 +0000 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-15 22:16:48.576767517 +0000 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-15 22:16:48.576767517 +0000 2024-04-15T22:20:17.843 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-15 22:16:48.576767517 +0000 2024-04-15T22:20:17.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T22:20:17.911 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-15T22:20:17.911 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-15T22:20:17.911 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000187784 s, 2.7 MB/s 2024-04-15T22:20:17.912 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T22:20:17.968 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-15 22:19:37.588284996 +0000 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-15 22:16:48.927760701 +0000 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-15 22:16:48.927760701 +0000 2024-04-15T22:20:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-15 22:16:48.927760701 +0000 2024-04-15T22:20:18.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T22:20:18.091 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-15T22:20:18.091 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-15T22:20:18.091 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000187615 s, 2.7 MB/s 2024-04-15T22:20:18.093 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T22:20:18.149 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-15 22:19:37.588284996 +0000 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-15 22:16:49.278753885 +0000 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-15 22:16:49.278753885 +0000 2024-04-15T22:20:18.205 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-15 22:16:49.278753885 +0000 2024-04-15T22:20:18.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T22:20:18.272 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-15T22:20:18.272 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-15T22:20:18.272 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000193731 s, 2.6 MB/s 2024-04-15T22:20:18.273 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T22:20:18.331 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-04-15T22:20:18.331 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T22:20:18.350 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T22:20:18.351 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-04-15T22:20:18.409 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-04-15 22:20:12.638872585 +0000 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-04-15 22:16:45.697736238 +0000 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-04-15 22:16:45.697736238 +0000 2024-04-15T22:20:18.410 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-04-15 22:16:45.697736238 +0000 2024-04-15T22:20:18.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T22:20:18.482 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-04-15T22:20:18.482 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-04-15T22:20:18.482 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000254052 s, 2.0 MB/s 2024-04-15T22:20:18.483 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T22:20:18.542 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-04-15T22:20:18.600 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T22:20:18.600 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:18.600 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-15T22:20:18.600 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:18.600 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:18.601 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-04-15 22:20:12.639872562 +0000 2024-04-15T22:20:18.601 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-04-15 22:16:46.035728289 +0000 2024-04-15T22:20:18.601 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-04-15 22:16:46.035728289 +0000 2024-04-15T22:20:18.601 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-04-15 22:16:46.035728289 +0000 2024-04-15T22:20:18.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T22:20:18.670 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-04-15T22:20:18.670 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-04-15T22:20:18.670 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000215704 s, 2.4 MB/s 2024-04-15T22:20:18.671 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T22:20:18.731 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-04-15T22:20:18.787 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-04-15 22:20:12.639872562 +0000 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-04-15 22:16:46.384720082 +0000 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-04-15 22:16:46.384720082 +0000 2024-04-15T22:20:18.788 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-04-15 22:16:46.384720082 +0000 2024-04-15T22:20:18.788 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T22:20:18.855 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:18 smithi163 ceph-mon[27043]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:18.855 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:18.858 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-04-15T22:20:18.858 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-04-15T22:20:18.858 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000212745 s, 2.4 MB/s 2024-04-15T22:20:18.859 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T22:20:18.917 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-04-15 22:20:12.640872539 +0000 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-04-15 22:16:46.723712110 +0000 2024-04-15T22:20:18.975 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-04-15 22:16:46.723712110 +0000 2024-04-15T22:20:18.976 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-04-15 22:16:46.723712110 +0000 2024-04-15T22:20:18.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T22:20:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-04-15T22:20:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-04-15T22:20:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000210045 s, 2.4 MB/s 2024-04-15T22:20:19.045 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T22:20:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:18 smithi081 ceph-mon[24860]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:19.103 INFO:tasks.cephadm:Deploying osd.0 on smithi081 with /dev/vg_nvme/lv_4... 2024-04-15T22:20:19.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T22:20:21.034 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:20 smithi081 ceph-mon[24860]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:21.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:20 smithi163 ceph-mon[27043]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T22:20:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T22:20:21.900 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-15T22:20:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-15T22:20:21.900 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135487 s, 774 MB/s 2024-04-15T22:20:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-15T22:20:22.544 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_4 2024-04-15T22:20:22.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:22 smithi081 ceph-mon[24860]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:23.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:22 smithi163 ceph-mon[27043]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:24.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:24 smithi081 ceph-mon[24860]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:24.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:24 smithi163 ceph-mon[27043]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:27.041 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:26 smithi081 ceph-mon[24860]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:27.041 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:26 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:27.041 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:26 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:27.041 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:26 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:27.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:26 smithi163 ceph-mon[27043]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:27.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:26 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:27.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:26 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:27.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:26 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:27.879 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:27 smithi081 ceph-mon[24860]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:28.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:27 smithi163 ceph-mon[27043]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:28.977 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:28 smithi081 ceph-mon[24860]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:29.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:28 smithi163 ceph-mon[27043]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:30.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:30 smithi081 ceph-mon[24860]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:30.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:30.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:30.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:30.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:30.986 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:30 smithi163 ceph-mon[27043]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:30.986 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:30.986 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:30.986 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:30.986 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:31.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:31 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/3806048905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a88478d-6021-4cd6-89d7-6439d2cb93fb"}]: dispatch 2024-04-15T22:20:31.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:31 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/3806048905' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a88478d-6021-4cd6-89d7-6439d2cb93fb"}]': finished 2024-04-15T22:20:31.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:31 smithi163 ceph-mon[27043]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T22:20:31.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:31 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:31.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:31 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3806048905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a88478d-6021-4cd6-89d7-6439d2cb93fb"}]: dispatch 2024-04-15T22:20:31.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:31 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3806048905' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a88478d-6021-4cd6-89d7-6439d2cb93fb"}]': finished 2024-04-15T22:20:31.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:31 smithi081 ceph-mon[24860]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T22:20:31.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:31 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:32.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:32 smithi163 ceph-mon[27043]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:32.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:32 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/4079720781' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:32.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:32 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:32.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:32 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:32.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:32 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:33.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:32 smithi081 ceph-mon[24860]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:33.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:32 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/4079720781' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:33.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:32 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:33.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:32 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:33.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:32 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:34.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:33 smithi081 ceph-mon[24860]: Updating smithi163:/etc/ceph/ceph.conf 2024-04-15T22:20:34.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:33 smithi081 ceph-mon[24860]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:20:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:33 smithi163 ceph-mon[27043]: Updating smithi163:/etc/ceph/ceph.conf 2024-04-15T22:20:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:33 smithi163 ceph-mon[27043]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:20:35.030 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:34 smithi081 ceph-mon[24860]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:35.030 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:35.079 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:34 smithi163 ceph-mon[27043]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:35.079 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: Reconfiguring mon.smithi081 (unknown last config time)... 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:36.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:35 smithi081 ceph-mon[24860]: Reconfiguring daemon mon.smithi081 on smithi081 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: Reconfiguring mon.smithi081 (unknown last config time)... 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:36.093 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:35 smithi163 ceph-mon[27043]: Reconfiguring daemon mon.smithi081 on smithi081 2024-04-15T22:20:36.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:36 smithi163 ceph-mon[27043]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:36.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:36 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:36.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:36 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi081.wfvsja", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:20:36.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:36 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:20:36.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:36 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:36 smithi081 ceph-mon[24860]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:36 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:36 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi081.wfvsja", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:20:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:36 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:20:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:36 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:38.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:37 smithi081 ceph-mon[24860]: Reconfiguring mgr.smithi081.wfvsja (unknown last config time)... 2024-04-15T22:20:38.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:37 smithi081 ceph-mon[24860]: Reconfiguring daemon mgr.smithi081.wfvsja on smithi081 2024-04-15T22:20:38.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:37 smithi163 ceph-mon[27043]: Reconfiguring mgr.smithi081.wfvsja (unknown last config time)... 2024-04-15T22:20:38.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:37 smithi163 ceph-mon[27043]: Reconfiguring daemon mgr.smithi081.wfvsja on smithi081 2024-04-15T22:20:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:38 smithi081 ceph-mon[24860]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:38 smithi163 ceph-mon[27043]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:40.042 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:39 smithi081 ceph-mon[24860]: Reconfiguring alertmanager.smithi081 (dependencies changed)... 2024-04-15T22:20:40.043 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:39 smithi081 ceph-mon[24860]: Reconfiguring daemon alertmanager.smithi081 on smithi081 2024-04-15T22:20:40.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:39 smithi163 ceph-mon[27043]: Reconfiguring alertmanager.smithi081 (dependencies changed)... 2024-04-15T22:20:40.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:39 smithi163 ceph-mon[27043]: Reconfiguring daemon alertmanager.smithi081 on smithi081 2024-04-15T22:20:41.032 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:40 smithi081 ceph-mon[24860]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:41.032 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T22:20:41.032 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:41.032 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:40 smithi081 ceph-mon[24860]: Deploying daemon osd.0 on smithi081 2024-04-15T22:20:41.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:40 smithi163 ceph-mon[27043]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:41.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T22:20:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:40 smithi163 ceph-mon[27043]: Deploying daemon osd.0 on smithi081 2024-04-15T22:20:42.979 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:42 smithi081 ceph-mon[24860]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:42 smithi163 ceph-mon[27043]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:44.703 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:44 smithi081 ceph-mon[24860]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:44.704 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:44 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:44.704 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:44 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi081", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:20:44.704 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:44 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:44.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:44 smithi163 ceph-mon[27043]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:44.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:44 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:44.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:44 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi081", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:20:44.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:44 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:45.688 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:45 smithi081 ceph-mon[24860]: Reconfiguring crash.smithi081 (monmap changed)... 2024-04-15T22:20:45.689 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:45 smithi081 ceph-mon[24860]: Reconfiguring daemon crash.smithi081 on smithi081 2024-04-15T22:20:45.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:45 smithi163 ceph-mon[27043]: Reconfiguring crash.smithi081 (monmap changed)... 2024-04-15T22:20:45.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:45 smithi163 ceph-mon[27043]: Reconfiguring daemon crash.smithi081 on smithi081 2024-04-15T22:20:46.656 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:46 smithi081 ceph-mon[24860]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:46.732 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:46 smithi163 ceph-mon[27043]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:48.018 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:20:48.018 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:20:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:20:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:20:49.046 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:48 smithi081 ceph-mon[24860]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:48 smithi163 ceph-mon[27043]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:50.577 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:50.577 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:50 smithi081 ceph-mon[24860]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:50.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:50.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:50 smithi163 ceph-mon[27043]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:51.692 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:51 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:51.692 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:51 smithi081 ceph-mon[24860]: Reconfiguring grafana.smithi081 (dependencies changed)... 2024-04-15T22:20:51.692 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:51 smithi081 ceph-mon[24860]: Reconfiguring daemon grafana.smithi081 on smithi081 2024-04-15T22:20:51.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:51 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:51.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:51 smithi163 ceph-mon[27043]: Reconfiguring grafana.smithi081 (dependencies changed)... 2024-04-15T22:20:51.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:51 smithi163 ceph-mon[27043]: Reconfiguring daemon grafana.smithi081 on smithi081 2024-04-15T22:20:52.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:52 smithi081 ceph-mon[24860]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:52.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:52 smithi081 ceph-mon[24860]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T22:20:52.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:52 smithi163 ceph-mon[27043]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:52.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:52 smithi163 ceph-mon[27043]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T22:20:53.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:53 smithi081 ceph-mon[24860]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T22:20:53.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:53 smithi081 ceph-mon[24860]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T22:20:53.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:53 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:53.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:53 smithi081 ceph-mon[24860]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:20:53.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:53 smithi163 ceph-mon[27043]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T22:20:53.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:53 smithi163 ceph-mon[27043]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T22:20:53.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:53 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:53.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:53 smithi163 ceph-mon[27043]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:20:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:54 smithi081 ceph-mon[24860]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:20:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:54 smithi081 ceph-mon[24860]: osdmap e8: 1 total, 0 up, 1 in 2024-04-15T22:20:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:54.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:54 smithi081 ceph-mon[24860]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:54.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:54 smithi163 ceph-mon[27043]: from='osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:20:54.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:54 smithi163 ceph-mon[27043]: osdmap e8: 1 total, 0 up, 1 in 2024-04-15T22:20:54.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:54 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:54.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:54 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:54.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:54 smithi163 ceph-mon[27043]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:20:55.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:55 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:20:55.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:55 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:20:55.732 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:55 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:55.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:55 smithi081 ceph-mon[24860]: osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866] boot 2024-04-15T22:20:55.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:55 smithi081 ceph-mon[24860]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T22:20:55.733 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:55 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:55.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:55 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:20:55.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:55 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:20:55.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:55 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:55.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:55 smithi163 ceph-mon[27043]: osd.0 [v2:172.21.15.81:6802/3069676866,v1:172.21.15.81:6803/3069676866] boot 2024-04-15T22:20:55.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:55 smithi163 ceph-mon[27043]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T22:20:55.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:55 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:20:56.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:56 smithi163 ceph-mon[27043]: osdmap e10: 1 total, 1 up, 1 in 2024-04-15T22:20:56.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:56.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:56 smithi163 ceph-mon[27043]: Reconfiguring crash.smithi163 (monmap changed)... 2024-04-15T22:20:56.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:20:56.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:56.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:56 smithi163 ceph-mon[27043]: Reconfiguring daemon crash.smithi163 on smithi163 2024-04-15T22:20:56.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:56 smithi163 ceph-mon[27043]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:56.683 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:56 smithi081 ceph-mon[24860]: osdmap e10: 1 total, 1 up, 1 in 2024-04-15T22:20:56.683 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:56.683 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:56 smithi081 ceph-mon[24860]: Reconfiguring crash.smithi163 (monmap changed)... 2024-04-15T22:20:56.683 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:20:56.683 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:56.683 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:56 smithi081 ceph-mon[24860]: Reconfiguring daemon crash.smithi163 on smithi163 2024-04-15T22:20:56.684 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:56 smithi081 ceph-mon[24860]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:58.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:58.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:57 smithi163 ceph-mon[27043]: Reconfiguring mgr.smithi163.cdbjip (monmap changed)... 2024-04-15T22:20:58.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.cdbjip", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:20:58.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:20:58.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:58.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:57 smithi163 ceph-mon[27043]: Reconfiguring daemon mgr.smithi163.cdbjip on smithi163 2024-04-15T22:20:58.214 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:58.214 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:57 smithi081 ceph-mon[24860]: Reconfiguring mgr.smithi163.cdbjip (monmap changed)... 2024-04-15T22:20:58.214 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.cdbjip", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:20:58.214 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:20:58.214 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:58.214 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:57 smithi081 ceph-mon[24860]: Reconfiguring daemon mgr.smithi163.cdbjip on smithi163 2024-04-15T22:20:58.251 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 0 on host 'smithi081' 2024-04-15T22:20:58.816 DEBUG:teuthology.orchestra.run.smithi081:osd.0> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.0.service 2024-04-15T22:20:58.819 INFO:tasks.cephadm:Deploying osd.1 on smithi081 with /dev/vg_nvme/lv_3... 2024-04-15T22:20:58.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T22:20:59.035 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:58 smithi163 ceph-mon[27043]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:59.035 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.035 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.036 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:20:59.036 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:20:59.036 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:59.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:58 smithi081 ceph-mon[24860]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:59.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:20:59.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:20:59.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: Reconfiguring mon.smithi163 (monmap changed)... 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: Reconfiguring daemon mon.smithi163 on smithi163 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.81:9093"}]: dispatch 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.81:3000"}]: dispatch 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.81:9095"}]: dispatch 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:20:59.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:59.929 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:59.929 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:20:59 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:00.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: Reconfiguring mon.smithi163 (monmap changed)... 2024-04-15T22:21:00.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: Reconfiguring daemon mon.smithi163 on smithi163 2024-04-15T22:21:00.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:00.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.81:9093"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.81:3000"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.81:9095"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:00.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:20:59 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:00.881 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T22:21:00.881 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T22:21:00.881 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-15T22:21:00.881 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-15T22:21:00.881 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01175 s, 892 MB/s 2024-04-15T22:21:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-15T22:21:01.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:00 smithi163 ceph-mon[27043]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:01.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:00 smithi163 ceph-mon[27043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:21:01.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:00 smithi163 ceph-mon[27043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.81:9093"}]: dispatch 2024-04-15T22:21:01.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:00 smithi163 ceph-mon[27043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:21:01.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:00 smithi163 ceph-mon[27043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.81:3000"}]: dispatch 2024-04-15T22:21:01.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:00 smithi163 ceph-mon[27043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:21:01.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:00 smithi163 ceph-mon[27043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.81:9095"}]: dispatch 2024-04-15T22:21:01.266 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:00 smithi081 ceph-mon[24860]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:01.266 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:00 smithi081 ceph-mon[24860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:21:01.266 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:00 smithi081 ceph-mon[24860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.81:9093"}]: dispatch 2024-04-15T22:21:01.266 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:00 smithi081 ceph-mon[24860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:21:01.266 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:00 smithi081 ceph-mon[24860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.81:3000"}]: dispatch 2024-04-15T22:21:01.266 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:00 smithi081 ceph-mon[24860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:21:01.266 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:00 smithi081 ceph-mon[24860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.81:9095"}]: dispatch 2024-04-15T22:21:01.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_3 2024-04-15T22:21:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:02 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:02 smithi163 ceph-mon[27043]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:02.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:02 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:02.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:02 smithi081 ceph-mon[24860]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:04.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:04 smithi081 ceph-mon[24860]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:04.717 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:04 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:04.718 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:04 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:04.718 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:04 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:04.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:04 smithi163 ceph-mon[27043]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:04.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:04 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:04.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:04 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:04.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:04 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:05.710 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:05 smithi081 ceph-mon[24860]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:05.828 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:05 smithi163 ceph-mon[27043]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:06.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:06 smithi081 ceph-mon[24860]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:06.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:06 smithi163 ceph-mon[27043]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:07.751 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:07 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2706702642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40fc7f9b-5bec-4774-ba1e-f5be50b9f05c"}]: dispatch 2024-04-15T22:21:07.751 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:07 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2706702642' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40fc7f9b-5bec-4774-ba1e-f5be50b9f05c"}]': finished 2024-04-15T22:21:07.751 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:07 smithi081 ceph-mon[24860]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T22:21:07.751 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:07 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:07.929 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:07 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2706702642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40fc7f9b-5bec-4774-ba1e-f5be50b9f05c"}]: dispatch 2024-04-15T22:21:07.929 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:07 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2706702642' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40fc7f9b-5bec-4774-ba1e-f5be50b9f05c"}]': finished 2024-04-15T22:21:07.929 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:07 smithi163 ceph-mon[27043]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T22:21:07.930 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:07 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:09.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:08 smithi081 ceph-mon[24860]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:09.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:09.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:08 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/627497945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:09.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:08 smithi163 ceph-mon[27043]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:08 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/627497945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:11.010 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:10 smithi081 ceph-mon[24860]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:11.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:10 smithi163 ceph-mon[27043]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:12.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:12.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:12.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:12.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:12.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:12.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:12.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:12.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:13.023 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:13.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:13.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:13.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:13.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:13.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:13.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:13.024 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:14.022 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:13 smithi081 ceph-mon[24860]: Detected new or changed devices on smithi081 2024-04-15T22:21:14.022 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:13 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi081 to 12864M 2024-04-15T22:21:14.109 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:13 smithi163 ceph-mon[27043]: Detected new or changed devices on smithi081 2024-04-15T22:21:14.109 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:13 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi081 to 12864M 2024-04-15T22:21:15.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:14 smithi081 ceph-mon[24860]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:15.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:14 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T22:21:15.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:14 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:15.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:14 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:15.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:14 smithi163 ceph-mon[27043]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:15.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:14 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T22:21:15.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:14 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:15.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:14 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:16.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:15 smithi081 ceph-mon[24860]: Deploying daemon osd.1 on smithi081 2024-04-15T22:21:16.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:15 smithi163 ceph-mon[27043]: Deploying daemon osd.1 on smithi081 2024-04-15T22:21:16.987 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:16 smithi081 ceph-mon[24860]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:17.062 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:16 smithi163 ceph-mon[27043]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:18 smithi081 ceph-mon[24860]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:19.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:18 smithi163 ceph-mon[27043]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:20.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:20 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:20.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:20 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:20.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:20 smithi081 ceph-mon[24860]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:20.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:20 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:20.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:20 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:20.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:20 smithi163 ceph-mon[27043]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:22.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:22 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:22.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:22 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:22.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:22 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:22.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:22 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:22.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:22 smithi163 ceph-mon[27043]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:22.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:22 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:22.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:22 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:22.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:22 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:22.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:22 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:22.976 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:22 smithi081 ceph-mon[24860]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:24.610 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:24 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:24.611 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:24 smithi081 ceph-mon[24860]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:24.641 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 1 on host 'smithi081' 2024-04-15T22:21:24.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:24 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:24.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:24 smithi163 ceph-mon[27043]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:25.239 DEBUG:teuthology.orchestra.run.smithi081:osd.1> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.1.service 2024-04-15T22:21:25.242 INFO:tasks.cephadm:Deploying osd.2 on smithi081 with /dev/vg_nvme/lv_2... 2024-04-15T22:21:25.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T22:21:25.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:25.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:25 smithi163 ceph-mon[27043]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:21:26.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:26.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:25 smithi081 ceph-mon[24860]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:21:26.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:26 smithi163 ceph-mon[27043]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T22:21:26.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:26 smithi163 ceph-mon[27043]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T22:21:26.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:26 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:26.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:26 smithi163 ceph-mon[27043]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:21:26.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:26 smithi163 ceph-mon[27043]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:26.982 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:26 smithi081 ceph-mon[24860]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T22:21:26.982 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:26 smithi081 ceph-mon[24860]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T22:21:26.982 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:26 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:26.982 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:26 smithi081 ceph-mon[24860]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:21:26.982 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:26 smithi081 ceph-mon[24860]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:26.983 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:26 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1[39392]: 2024-04-15T22:21:26.649+0000 7f063b824700 -1 osd.1 0 waiting for initial osdmap 2024-04-15T22:21:26.983 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:26 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1[39392]: 2024-04-15T22:21:26.652+0000 7f06371bd700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:21:27.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:27 smithi163 ceph-mon[27043]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:21:27.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:27 smithi163 ceph-mon[27043]: osdmap e13: 2 total, 1 up, 2 in 2024-04-15T22:21:27.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:27 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:27.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:27 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:27.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:27 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:28.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:27 smithi081 ceph-mon[24860]: from='osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:21:28.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:27 smithi081 ceph-mon[24860]: osdmap e13: 2 total, 1 up, 2 in 2024-04-15T22:21:28.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:27 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:28.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:27 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:28.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:27 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:28.586 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T22:21:28.587 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T22:21:28.587 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-15T22:21:28.587 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-15T22:21:28.587 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0153585 s, 683 MB/s 2024-04-15T22:21:28.587 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-15T22:21:28.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:28 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:21:28.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:28 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:21:28.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:28 smithi081 ceph-mon[24860]: osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360] boot 2024-04-15T22:21:28.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:28 smithi081 ceph-mon[24860]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T22:21:28.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:28 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:28.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:28 smithi081 ceph-mon[24860]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:28.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:28 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:21:28.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:28 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:21:28.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:28 smithi163 ceph-mon[27043]: osd.1 [v2:172.21.15.81:6810/2281467360,v1:172.21.15.81:6811/2281467360] boot 2024-04-15T22:21:28.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:28 smithi163 ceph-mon[27043]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T22:21:28.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:28 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:21:28.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:28 smithi163 ceph-mon[27043]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:21:29.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_2 2024-04-15T22:21:29.922 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:29 smithi081 ceph-mon[24860]: osdmap e15: 2 total, 2 up, 2 in 2024-04-15T22:21:29.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:29 smithi163 ceph-mon[27043]: osdmap e15: 2 total, 2 up, 2 in 2024-04-15T22:21:30.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:30 smithi163 ceph-mon[27043]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:30.935 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:30 smithi081 ceph-mon[24860]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:31.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:31 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:31.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:31 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:31.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:31 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:31.961 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:31 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:31.961 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:31 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:31.961 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:31 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:33.107 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:32 smithi081 ceph-mon[24860]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:33.107 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:32 smithi081 ceph-mon[24860]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:33.107 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:32 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:33.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:32 smithi163 ceph-mon[27043]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:33.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:32 smithi163 ceph-mon[27043]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:33.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:32 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:34.517 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:34 smithi081 ceph-mon[24860]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:34.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:34 smithi163 ceph-mon[27043]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:35 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1741145136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0091596-5d84-461a-ba14-6c900cc9af14"}]: dispatch 2024-04-15T22:21:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:35 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1741145136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0091596-5d84-461a-ba14-6c900cc9af14"}]': finished 2024-04-15T22:21:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:35 smithi081 ceph-mon[24860]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T22:21:35.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:35.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:35 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2026303978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:35.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:35 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1741145136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0091596-5d84-461a-ba14-6c900cc9af14"}]: dispatch 2024-04-15T22:21:35.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:35 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1741145136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0091596-5d84-461a-ba14-6c900cc9af14"}]': finished 2024-04-15T22:21:35.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:35 smithi163 ceph-mon[27043]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T22:21:35.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:35.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:35 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2026303978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:36.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:36 smithi081 ceph-mon[24860]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:36.844 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:36 smithi163 ceph-mon[27043]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:39.033 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:38 smithi081 ceph-mon[24860]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:38 smithi163 ceph-mon[27043]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:40.973 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:40 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:41.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:41.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:40 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:41.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:41 smithi081 ceph-mon[24860]: Detected new or changed devices on smithi081 2024-04-15T22:21:41.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:41 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi081 to 6432M 2024-04-15T22:21:42.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:41 smithi163 ceph-mon[27043]: Detected new or changed devices on smithi081 2024-04-15T22:21:42.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:41 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi081 to 6432M 2024-04-15T22:21:43.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:42 smithi081 ceph-mon[24860]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:43.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:42 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T22:21:43.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:42 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:42 smithi163 ceph-mon[27043]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:42 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T22:21:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:42 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:43.847 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:43 smithi081 ceph-mon[24860]: Deploying daemon osd.2 on smithi081 2024-04-15T22:21:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:43 smithi163 ceph-mon[27043]: Deploying daemon osd.2 on smithi081 2024-04-15T22:21:44.901 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:44 smithi081 ceph-mon[24860]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:45.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:44 smithi163 ceph-mon[27043]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:46.996 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:46 smithi081 ceph-mon[24860]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:47.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:46 smithi163 ceph-mon[27043]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:48.677 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:48.677 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:48.678 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:48 smithi081 ceph-mon[24860]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:48.678 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:21:48.678 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:21:48.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:48.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:48.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:48 smithi163 ceph-mon[27043]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:48.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:21:48.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:21:50.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:50.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:50.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:50.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:50.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:50 smithi081 ceph-mon[24860]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:50.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:50.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:50.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:50.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:50.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:50 smithi163 ceph-mon[27043]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:52.775 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 2 on host 'smithi081' 2024-04-15T22:21:53.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:52 smithi081 ceph-mon[24860]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:53.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:52 smithi081 ceph-mon[24860]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:21:53.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:52 smithi163 ceph-mon[27043]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:53.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:52 smithi163 ceph-mon[27043]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:21:53.433 DEBUG:teuthology.orchestra.run.smithi081:osd.2> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.2.service 2024-04-15T22:21:53.436 INFO:tasks.cephadm:Deploying osd.3 on smithi081 with /dev/vg_nvme/lv_1... 2024-04-15T22:21:53.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T22:21:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:53 smithi081 ceph-mon[24860]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T22:21:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:53 smithi081 ceph-mon[24860]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T22:21:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:53 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:53 smithi081 ceph-mon[24860]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:21:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:53 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:54.055 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:53 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2[43080]: 2024-04-15T22:21:53.726+0000 7fe631289700 -1 osd.2 0 waiting for initial osdmap 2024-04-15T22:21:54.055 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:53 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2[43080]: 2024-04-15T22:21:53.734+0000 7fe62cc22700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:21:54.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:53 smithi163 ceph-mon[27043]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T22:21:54.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:53 smithi163 ceph-mon[27043]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T22:21:54.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:53 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:54.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:53 smithi163 ceph-mon[27043]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:21:54.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:53 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:54 smithi081 ceph-mon[24860]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:54 smithi081 ceph-mon[24860]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:21:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:54 smithi081 ceph-mon[24860]: osdmap e18: 3 total, 2 up, 3 in 2024-04-15T22:21:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:55.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:54 smithi163 ceph-mon[27043]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:55.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:54 smithi163 ceph-mon[27043]: from='osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:21:55.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:54 smithi163 ceph-mon[27043]: osdmap e18: 3 total, 2 up, 3 in 2024-04-15T22:21:55.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:54 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:55.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:54 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:56.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:55 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:21:56.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:55 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:21:56.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:55 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:56.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:55 smithi081 ceph-mon[24860]: osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695] boot 2024-04-15T22:21:56.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:55 smithi081 ceph-mon[24860]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T22:21:56.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:55 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:56.106 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T22:21:56.106 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T22:21:56.107 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-15T22:21:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-15T22:21:56.107 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126126 s, 831 MB/s 2024-04-15T22:21:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-15T22:21:56.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:55 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:21:56.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:55 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:21:56.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:55 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:56.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:55 smithi163 ceph-mon[27043]: osd.2 [v2:172.21.15.81:6818/2961844695,v1:172.21.15.81:6819/2961844695] boot 2024-04-15T22:21:56.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:55 smithi163 ceph-mon[27043]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T22:21:56.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:55 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:21:56.811 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_1 2024-04-15T22:21:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:56 smithi081 ceph-mon[24860]: pgmap v71: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T22:21:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T22:21:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:56 smithi081 ceph-mon[24860]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T22:21:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:21:57.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:56 smithi163 ceph-mon[27043]: pgmap v71: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:21:57.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T22:21:57.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T22:21:57.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:56 smithi163 ceph-mon[27043]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T22:21:57.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:21:58.199 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T22:21:58.200 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:57 smithi081 ceph-mon[24860]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T22:21:58.200 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:57 smithi081 ceph-mon[24860]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:21:58.200 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:21:57 smithi081 sudo[44334]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:21:58.200 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:21:57 smithi081 sudo[44334]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:21:58.200 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:21:57 smithi081 sudo[44334]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:21:58.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T22:21:58.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:57 smithi163 ceph-mon[27043]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T22:21:58.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:57 smithi163 ceph-mon[27043]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:21:58.477 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44348]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:21:58.477 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44348]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:21:58.477 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44348]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:21:58.741 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44377]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:21:58.741 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44377]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:21:58.741 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44377]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:21:58.996 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44404]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:21:58.996 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44404]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:21:58.996 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44404]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:21:58.997 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:58 smithi081 ceph-mon[24860]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T22:21:58.997 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:58.997 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:58 smithi081 ceph-mon[24860]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:58.997 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:58.997 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:58.997 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:59.262 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44464]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:21:59.262 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:58 smithi081 sudo[44464]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:21:59.262 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44464]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:21:59.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:58 smithi163 ceph-mon[27043]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T22:21:59.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:21:59.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:58 smithi163 ceph-mon[27043]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:59.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:59.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:59.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:21:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:59.517 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44483]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:21:59.517 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44483]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:21:59.517 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44483]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:21:59.777 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44558]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:21:59.777 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44558]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:21:59.777 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44558]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:22:00.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44590]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:22:00.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44590]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:22:00.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:21:59 smithi081 sudo[44590]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:22:00.276 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 sudo[29365]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:22:00.276 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 sudo[29365]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:22:00.276 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 sudo[29365]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: mgrmap e19: smithi081.wfvsja(active, since 2m), standbys: smithi163.cdbjip 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:22:00.416 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:00 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:22:00.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 sudo[29368]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 sudo[29368]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 sudo[29368]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: mgrmap e19: smithi081.wfvsja(active, since 2m), standbys: smithi163.cdbjip 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi081"}]: dispatch 2024-04-15T22:22:00.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:00 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-04-15T22:22:01.347 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:01 smithi081 ceph-mon[24860]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:22:01.348 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:01 smithi081 ceph-mon[24860]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:22:01.348 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:01 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:01.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:01 smithi163 ceph-mon[27043]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:22:01.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:01 smithi163 ceph-mon[27043]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:22:01.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:01 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:02 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:02 smithi163 ceph-mon[27043]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:02 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1280898623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38f7a45b-9e4a-4a7f-8a58-18b0818db1f3"}]: dispatch 2024-04-15T22:22:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:02 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1280898623' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38f7a45b-9e4a-4a7f-8a58-18b0818db1f3"}]': finished 2024-04-15T22:22:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:02 smithi163 ceph-mon[27043]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T22:22:02.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:02 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:02.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:02 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:02.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:02 smithi081 ceph-mon[24860]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:02.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:02 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1280898623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38f7a45b-9e4a-4a7f-8a58-18b0818db1f3"}]: dispatch 2024-04-15T22:22:02.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:02 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1280898623' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38f7a45b-9e4a-4a7f-8a58-18b0818db1f3"}]': finished 2024-04-15T22:22:02.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:02 smithi081 ceph-mon[24860]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T22:22:02.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:02 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:03.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:03 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1460593901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:03.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:03 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1460593901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:04.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:04 smithi081 ceph-mon[24860]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-15T22:22:04.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:04 smithi163 ceph-mon[27043]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-15T22:22:06.978 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:06 smithi163 ceph-mon[27043]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-15T22:22:07.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:06 smithi081 ceph-mon[24860]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-15T22:22:09.049 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-15T22:22:09.049 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.049 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:09.050 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:08 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:09.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:08 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:09.977 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:09 smithi081 ceph-mon[24860]: Detected new or changed devices on smithi081 2024-04-15T22:22:09.977 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:09 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi081 to 4288M 2024-04-15T22:22:09.977 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:09 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T22:22:09.977 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:09 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:10.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:09 smithi163 ceph-mon[27043]: Detected new or changed devices on smithi081 2024-04-15T22:22:10.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:09 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi081 to 4288M 2024-04-15T22:22:10.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:09 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T22:22:10.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:09 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:11.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:10 smithi081 ceph-mon[24860]: Deploying daemon osd.3 on smithi081 2024-04-15T22:22:11.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:10 smithi081 ceph-mon[24860]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-15T22:22:11.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:10 smithi163 ceph-mon[27043]: Deploying daemon osd.3 on smithi081 2024-04-15T22:22:11.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:10 smithi163 ceph-mon[27043]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-15T22:22:13.046 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:12 smithi081 ceph-mon[24860]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-15T22:22:13.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:12 smithi163 ceph-mon[27043]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-15T22:22:14.553 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:14 smithi081 ceph-mon[24860]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 68 KiB/s wr, 4 op/s 2024-04-15T22:22:14.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:14 smithi163 ceph-mon[27043]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 68 KiB/s wr, 4 op/s 2024-04-15T22:22:16.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:15 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:16.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:15 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:16.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:15 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:16.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:15 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:16.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:15 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:16.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:15 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:16.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:15 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:16.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:15 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:17.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:16 smithi081 ceph-mon[24860]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:17.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:16 smithi163 ceph-mon[27043]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:18.716 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 3 on host 'smithi081' 2024-04-15T22:22:19.038 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:18 smithi081 ceph-mon[24860]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:19.039 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:18 smithi081 ceph-mon[24860]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:22:19.039 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:19.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:18 smithi163 ceph-mon[27043]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:19.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:18 smithi163 ceph-mon[27043]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:22:19.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:19.336 DEBUG:teuthology.orchestra.run.smithi081:osd.3> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.3.service 2024-04-15T22:22:19.339 INFO:tasks.cephadm:Deploying osd.4 on smithi163 with /dev/vg_nvme/lv_4... 2024-04-15T22:22:19.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T22:22:20.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:19 smithi163 ceph-mon[27043]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T22:22:20.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:19 smithi163 ceph-mon[27043]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T22:22:20.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:19 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:20.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:19 smithi163 ceph-mon[27043]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:22:20.277 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:19 smithi081 ceph-mon[24860]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T22:22:20.277 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:19 smithi081 ceph-mon[24860]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T22:22:20.277 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:19 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:20.278 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:19 smithi081 ceph-mon[24860]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-15T22:22:20.278 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:22:19 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3[46351]: 2024-04-15T22:22:19.909+0000 7fbf97d0d700 -1 osd.3 0 waiting for initial osdmap 2024-04-15T22:22:20.278 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:22:19 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3[46351]: 2024-04-15T22:22:19.913+0000 7fbf90ea1700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:22:21.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:20 smithi163 ceph-mon[27043]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:21.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:20 smithi163 ceph-mon[27043]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:22:21.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:20 smithi163 ceph-mon[27043]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T22:22:21.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:20 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:21.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:20 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:21.222 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:20 smithi081 ceph-mon[24860]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:21.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:20 smithi081 ceph-mon[24860]: from='osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-15T22:22:21.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:20 smithi081 ceph-mon[24860]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T22:22:21.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:20 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:21.223 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:20 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:21.315 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T22:22:21.315 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T22:22:21.315 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-04-15T22:22:21.315 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-04-15T22:22:21.315 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113294 s, 926 MB/s 2024-04-15T22:22:21.315 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-04-15T22:22:21.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-04-15T22:22:22.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:21 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:22:22.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:21 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:22:22.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:22.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:21 smithi163 ceph-mon[27043]: osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015] boot 2024-04-15T22:22:22.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:21 smithi163 ceph-mon[27043]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T22:22:22.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:22.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:22.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:21 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:22:22.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:21 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:22:22.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:22.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:21 smithi081 ceph-mon[24860]: osd.3 [v2:172.21.15.81:6826/3454580015,v1:172.21.15.81:6827/3454580015] boot 2024-04-15T22:22:22.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:21 smithi081 ceph-mon[24860]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T22:22:22.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:22.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:23.180 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:22 smithi163 ceph-mon[27043]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:23.181 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:22 smithi163 ceph-mon[27043]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T22:22:23.248 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:22 smithi081 ceph-mon[24860]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:22:23.249 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:22 smithi081 ceph-mon[24860]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T22:22:24.667 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:24 smithi081 ceph-mon[24860]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:24.667 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:24 smithi081 ceph-mon[24860]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:24.667 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:24 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:22:24.667 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:24 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:22:24.667 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:24 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:24.889 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:24 smithi163 ceph-mon[27043]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:24.889 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:24 smithi163 ceph-mon[27043]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:24.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:24 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:22:24.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:24 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:22:24.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:24 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi081 to 3216M 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:26.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:26.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:26.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:25 smithi081 ceph-mon[24860]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi081 to 3216M 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:26.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:25 smithi163 ceph-mon[27043]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:27.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:27 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/1363885178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "801b8f37-6b69-4960-9f37-caecbf0f6113"}]: dispatch 2024-04-15T22:22:27.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:27 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "801b8f37-6b69-4960-9f37-caecbf0f6113"}]: dispatch 2024-04-15T22:22:27.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:27 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "801b8f37-6b69-4960-9f37-caecbf0f6113"}]': finished 2024-04-15T22:22:27.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:27 smithi081 ceph-mon[24860]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T22:22:27.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:27 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:27.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:27 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/2519453682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:27.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:27 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/1363885178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "801b8f37-6b69-4960-9f37-caecbf0f6113"}]: dispatch 2024-04-15T22:22:27.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:27 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "801b8f37-6b69-4960-9f37-caecbf0f6113"}]: dispatch 2024-04-15T22:22:27.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:27 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "801b8f37-6b69-4960-9f37-caecbf0f6113"}]': finished 2024-04-15T22:22:27.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:27 smithi163 ceph-mon[27043]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T22:22:27.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:27 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:27.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:27 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/2519453682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:28.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:28 smithi081 ceph-mon[24860]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:28.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:28 smithi163 ceph-mon[27043]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:30.847 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:30 smithi163 ceph-mon[27043]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:31.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:30 smithi081 ceph-mon[24860]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:32.873 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:32 smithi163 ceph-mon[27043]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:32.874 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:32 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T22:22:32.874 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:32 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:33.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:32 smithi081 ceph-mon[24860]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:33.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:32 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T22:22:33.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:32 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:33.863 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:33 smithi163 ceph-mon[27043]: Deploying daemon osd.4 on smithi163 2024-04-15T22:22:34.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:33 smithi081 ceph-mon[24860]: Deploying daemon osd.4 on smithi163 2024-04-15T22:22:34.938 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:34 smithi163 ceph-mon[27043]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:35.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:34 smithi081 ceph-mon[24860]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:36.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:36 smithi163 ceph-mon[27043]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:36 smithi081 ceph-mon[24860]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:38.519 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:38.519 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:38.519 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:38.519 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:38.519 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:38 smithi163 ceph-mon[27043]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:38.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:38.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:38.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:38.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:38.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:38 smithi081 ceph-mon[24860]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:40.737 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 4 on host 'smithi163' 2024-04-15T22:22:41.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:40 smithi081 ceph-mon[24860]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:41.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:40 smithi081 ceph-mon[24860]: from='osd.4 [v2:172.21.15.163:6800/902397810,v1:172.21.15.163:6801/902397810]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:22:41.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:40 smithi081 ceph-mon[24860]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:22:41.060 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:40 smithi163 ceph-mon[27043]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:41.060 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:40 smithi163 ceph-mon[27043]: from='osd.4 [v2:172.21.15.163:6800/902397810,v1:172.21.15.163:6801/902397810]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:22:41.060 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:40 smithi163 ceph-mon[27043]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:22:41.655 DEBUG:teuthology.orchestra.run.smithi163:osd.4> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.4.service 2024-04-15T22:22:41.657 INFO:tasks.cephadm:Deploying osd.5 on smithi163 with /dev/vg_nvme/lv_3... 2024-04-15T22:22:41.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T22:22:41.899 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-mon[27043]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T22:22:41.899 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-mon[27043]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T22:22:41.899 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:41.900 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-mon[27043]: from='osd.4 [v2:172.21.15.163:6800/902397810,v1:172.21.15.163:6801/902397810]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:22:41.900 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-mon[27043]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:22:41.900 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:41.900 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:41.900 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:22:41.739+0000 7fe1d371a700 -1 osd.4 0 waiting for initial osdmap 2024-04-15T22:22:41.900 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:22:41 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:22:41.744+0000 7fe1cf8b4700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:22:42.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:41 smithi081 ceph-mon[24860]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T22:22:42.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:41 smithi081 ceph-mon[24860]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T22:22:42.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:42.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:41 smithi081 ceph-mon[24860]: from='osd.4 [v2:172.21.15.163:6800/902397810,v1:172.21.15.163:6801/902397810]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:22:42.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:41 smithi081 ceph-mon[24860]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:22:42.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:42.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:42.847 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:42 smithi163 ceph-mon[27043]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:42.847 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:42 smithi163 ceph-mon[27043]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:22:42.847 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:42 smithi163 ceph-mon[27043]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T22:22:42.847 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:42 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:42.847 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:42 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:43.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:42 smithi081 ceph-mon[24860]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:22:43.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:42 smithi081 ceph-mon[24860]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:22:43.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:42 smithi081 ceph-mon[24860]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T22:22:43.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:42 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:43.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:42 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:43.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:43 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:22:43.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:43 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:22:43.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:43 smithi163 ceph-mon[27043]: osd.4 [v2:172.21.15.163:6800/902397810,v1:172.21.15.163:6801/902397810] boot 2024-04-15T22:22:43.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:43 smithi163 ceph-mon[27043]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T22:22:43.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:43 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:43.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:43 smithi163 ceph-mon[27043]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T22:22:44.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:43 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:22:44.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:43 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:22:44.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:43 smithi081 ceph-mon[24860]: osd.4 [v2:172.21.15.163:6800/902397810,v1:172.21.15.163:6801/902397810] boot 2024-04-15T22:22:44.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:43 smithi081 ceph-mon[24860]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T22:22:44.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:43 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:44.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:43 smithi081 ceph-mon[24860]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T22:22:44.502 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T22:22:44.502 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T22:22:44.502 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-04-15T22:22:44.502 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-04-15T22:22:44.502 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125981 s, 832 MB/s 2024-04-15T22:22:44.502 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-04-15T22:22:45.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:44 smithi081 ceph-mon[24860]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:22:45.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:44 smithi081 ceph-mon[24860]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T22:22:45.124 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:44 smithi163 ceph-mon[27043]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:22:45.124 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:44 smithi163 ceph-mon[27043]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T22:22:45.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-04-15T22:22:46.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T22:22:46.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: Detected new or changed devices on smithi163 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi163 to 15936M 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:46.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: Detected new or changed devices on smithi163 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi163 to 15936M 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:46.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:46.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:46.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:46.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:22:47.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:47 smithi163 ceph-mon[27043]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T22:22:47.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:47 smithi081 ceph-mon[24860]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T22:22:48.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:48 smithi081 ceph-mon[24860]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:22:48.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:48.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:48.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:48 smithi081 ceph-mon[24860]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:48.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:22:48.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:22:48.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:48.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:48 smithi163 ceph-mon[27043]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:22:48.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:48.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:48.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:48 smithi163 ceph-mon[27043]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:48.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:22:48.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:22:48.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:51.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:50 smithi081 ceph-mon[24860]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:51.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:50 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/2155438927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2727f5ae-a8e6-4677-a2a4-74c3c9810a86"}]: dispatch 2024-04-15T22:22:51.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:50 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2727f5ae-a8e6-4677-a2a4-74c3c9810a86"}]: dispatch 2024-04-15T22:22:51.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:50 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2727f5ae-a8e6-4677-a2a4-74c3c9810a86"}]': finished 2024-04-15T22:22:51.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:50 smithi081 ceph-mon[24860]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T22:22:51.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:22:51.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:50 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/580783003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:51.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:50 smithi163 ceph-mon[27043]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:51.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:50 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/2155438927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2727f5ae-a8e6-4677-a2a4-74c3c9810a86"}]: dispatch 2024-04-15T22:22:51.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:50 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2727f5ae-a8e6-4677-a2a4-74c3c9810a86"}]: dispatch 2024-04-15T22:22:51.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:50 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2727f5ae-a8e6-4677-a2a4-74c3c9810a86"}]': finished 2024-04-15T22:22:51.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:50 smithi163 ceph-mon[27043]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T22:22:51.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:22:51.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:50 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/580783003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:53.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:52 smithi081 ceph-mon[24860]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T22:22:53.109 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:52 smithi163 ceph-mon[27043]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T22:22:54.582 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:54 smithi163 ceph-mon[27043]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T22:22:54.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:54 smithi081 ceph-mon[24860]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T22:22:56.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:56 smithi163 ceph-mon[27043]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T22:22:56.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T22:22:56.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:56 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:56.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:56 smithi163 ceph-mon[27043]: Deploying daemon osd.5 on smithi163 2024-04-15T22:22:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:56 smithi081 ceph-mon[24860]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T22:22:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T22:22:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:56 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:57.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:56 smithi081 ceph-mon[24860]: Deploying daemon osd.5 on smithi163 2024-04-15T22:22:58.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:22:58 smithi163 ceph-mon[27043]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-15T22:22:59.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:22:58 smithi081 ceph-mon[24860]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-15T22:23:01.012 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:00 smithi163 ceph-mon[27043]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:01.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:00 smithi081 ceph-mon[24860]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:02.290 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:01 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:02.290 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:01 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:02.290 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:01 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:02.290 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:01 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:02.290 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:01 smithi163 ceph-mon[27043]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:02.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:01 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:02.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:01 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:02.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:01 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:02.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:01 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:02.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:01 smithi081 ceph-mon[24860]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:04.332 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 5 on host 'smithi163' 2024-04-15T22:23:04.674 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:04 smithi163 ceph-mon[27043]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:04.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:04 smithi163 ceph-mon[27043]: from='osd.5 [v2:172.21.15.163:6808/4193470634,v1:172.21.15.163:6809/4193470634]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:23:04.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:04 smithi163 ceph-mon[27043]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:23:04.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:04 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:04.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:04 smithi081 ceph-mon[24860]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:04.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:04 smithi081 ceph-mon[24860]: from='osd.5 [v2:172.21.15.163:6808/4193470634,v1:172.21.15.163:6809/4193470634]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:23:04.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:04 smithi081 ceph-mon[24860]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:23:04.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:04 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:05.286 DEBUG:teuthology.orchestra.run.smithi163:osd.5> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.5.service 2024-04-15T22:23:05.288 INFO:tasks.cephadm:Deploying osd.6 on smithi163 with /dev/vg_nvme/lv_2... 2024-04-15T22:23:05.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T22:23:05.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-mon[27043]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T22:23:05.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-mon[27043]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T22:23:05.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:05.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-mon[27043]: from='osd.5 [v2:172.21.15.163:6808/4193470634,v1:172.21.15.163:6809/4193470634]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:05.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-mon[27043]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:05.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:05.641 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:23:05.483+0000 7f5c870c1700 -1 osd.5 0 waiting for initial osdmap 2024-04-15T22:23:05.641 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:23:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:23:05.488+0000 7f5c81a58700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:23:05.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:05 smithi081 ceph-mon[24860]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T22:23:05.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:05 smithi081 ceph-mon[24860]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T22:23:05.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:05 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:05.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:05 smithi081 ceph-mon[24860]: from='osd.5 [v2:172.21.15.163:6808/4193470634,v1:172.21.15.163:6809/4193470634]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:05.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:05 smithi081 ceph-mon[24860]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:05.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:05 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:06.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:06 smithi081 ceph-mon[24860]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:23:06.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:06 smithi081 ceph-mon[24860]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T22:23:06.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:06 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:06.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:06 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:06.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:06 smithi081 ceph-mon[24860]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:06.848 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:06 smithi163 ceph-mon[27043]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:23:06.848 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:06 smithi163 ceph-mon[27043]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T22:23:06.848 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:06 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:06.848 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:06 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:06.848 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:06 smithi163 ceph-mon[27043]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:23:07.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:07 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:23:07.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:07 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:23:07.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:07 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:07.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:07 smithi081 ceph-mon[24860]: osd.5 [v2:172.21.15.163:6808/4193470634,v1:172.21.15.163:6809/4193470634] boot 2024-04-15T22:23:07.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:07 smithi081 ceph-mon[24860]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T22:23:07.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:07 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:07.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:07 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:23:07.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:07 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:23:07.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:07 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:07.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:07 smithi163 ceph-mon[27043]: osd.5 [v2:172.21.15.163:6808/4193470634,v1:172.21.15.163:6809/4193470634] boot 2024-04-15T22:23:07.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:07 smithi163 ceph-mon[27043]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T22:23:07.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:07 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:23:08.671 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T22:23:08.671 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T22:23:08.672 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-04-15T22:23:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-04-15T22:23:08.672 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112144 s, 935 MB/s 2024-04-15T22:23:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-04-15T22:23:08.798 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:08 smithi163 ceph-mon[27043]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T22:23:08.798 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:08 smithi163 ceph-mon[27043]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:08.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:08 smithi081 ceph-mon[24860]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T22:23:08.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:08 smithi081 ceph-mon[24860]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:09.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-04-15T22:23:09.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:09 smithi163 ceph-mon[27043]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T22:23:09.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:09 smithi081 ceph-mon[24860]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: Detected new or changed devices on smithi163 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi163 to 7968M 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:10.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:10.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:10.806 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:10 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: Detected new or changed devices on smithi163 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi163 to 7968M 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:10.887 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:10.888 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:10.888 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:10.888 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:10.888 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:10 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:11.765 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:11 smithi163 ceph-mon[27043]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T22:23:11.765 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:11 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:23:11.765 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:11 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:23:11.765 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:11 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:11.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:11 smithi081 ceph-mon[24860]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T22:23:11.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:11 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:23:11.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:11 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:23:11.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:11 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:12.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:12 smithi163 ceph-mon[27043]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:23:12.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:12 smithi163 ceph-mon[27043]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:12.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:12 smithi081 ceph-mon[24860]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:23:12.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:12 smithi081 ceph-mon[24860]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:14.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:14 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/1329772262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd789380-c4f0-4eef-a592-d3819599d2b3"}]: dispatch 2024-04-15T22:23:14.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:14 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd789380-c4f0-4eef-a592-d3819599d2b3"}]: dispatch 2024-04-15T22:23:14.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:14 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd789380-c4f0-4eef-a592-d3819599d2b3"}]': finished 2024-04-15T22:23:14.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:14 smithi081 ceph-mon[24860]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T22:23:14.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:14 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:14.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:14 smithi081 ceph-mon[24860]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:14.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:14 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/1939628971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:23:14.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:14 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/1329772262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd789380-c4f0-4eef-a592-d3819599d2b3"}]: dispatch 2024-04-15T22:23:14.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:14 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd789380-c4f0-4eef-a592-d3819599d2b3"}]: dispatch 2024-04-15T22:23:14.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:14 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd789380-c4f0-4eef-a592-d3819599d2b3"}]': finished 2024-04-15T22:23:14.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:14 smithi163 ceph-mon[27043]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T22:23:14.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:14 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:14.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:14 smithi163 ceph-mon[27043]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:14.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:14 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/1939628971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:23:17.019 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:16 smithi163 ceph-mon[27043]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:17.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:16 smithi081 ceph-mon[24860]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:18.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:18 smithi163 ceph-mon[27043]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:19.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:18 smithi081 ceph-mon[24860]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:20.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:20 smithi163 ceph-mon[27043]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:20.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:20 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T22:23:20.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:20 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:20.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:20 smithi163 ceph-mon[27043]: Deploying daemon osd.6 on smithi163 2024-04-15T22:23:21.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:20 smithi081 ceph-mon[24860]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:21.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:20 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T22:23:21.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:20 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:21.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:20 smithi081 ceph-mon[24860]: Deploying daemon osd.6 on smithi163 2024-04-15T22:23:22.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:22 smithi163 ceph-mon[27043]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:23.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:22 smithi081 ceph-mon[24860]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:24.590 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:24 smithi163 ceph-mon[27043]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:24.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:24 smithi081 ceph-mon[24860]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:26.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:26.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:26.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:26.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:25 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:26.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:26.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:25 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:27.140 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:26 smithi163 ceph-mon[27043]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:27.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:26 smithi081 ceph-mon[24860]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:28.444 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 6 on host 'smithi163' 2024-04-15T22:23:28.957 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:28 smithi163 ceph-mon[27043]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:28.957 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:28 smithi163 ceph-mon[27043]: from='osd.6 [v2:172.21.15.163:6816/2316852861,v1:172.21.15.163:6817/2316852861]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:23:28.957 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:28 smithi163 ceph-mon[27043]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:23:28.957 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:28 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:29.255 DEBUG:teuthology.orchestra.run.smithi163:osd.6> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.6.service 2024-04-15T22:23:29.257 INFO:tasks.cephadm:Deploying osd.7 on smithi163 with /dev/vg_nvme/lv_1... 2024-04-15T22:23:29.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T22:23:29.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:28 smithi081 ceph-mon[24860]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:29.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:28 smithi081 ceph-mon[24860]: from='osd.6 [v2:172.21.15.163:6816/2316852861,v1:172.21.15.163:6817/2316852861]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:23:29.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:28 smithi081 ceph-mon[24860]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:23:29.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:28 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:30.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-mon[27043]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T22:23:30.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-mon[27043]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T22:23:30.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:30.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-mon[27043]: from='osd.6 [v2:172.21.15.163:6816/2316852861,v1:172.21.15.163:6817/2316852861]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:30.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-mon[27043]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:30.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:30.178 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:23:29.874+0000 7fabbc341700 -1 osd.6 0 waiting for initial osdmap 2024-04-15T22:23:30.178 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:23:29 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:23:29.880+0000 7fabb4cd4700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:23:30.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:29 smithi081 ceph-mon[24860]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T22:23:30.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:29 smithi081 ceph-mon[24860]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T22:23:30.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:29 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:30.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:29 smithi081 ceph-mon[24860]: from='osd.6 [v2:172.21.15.163:6816/2316852861,v1:172.21.15.163:6817/2316852861]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:30.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:29 smithi081 ceph-mon[24860]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:30.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:29 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:31.135 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:30 smithi163 ceph-mon[27043]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:31.135 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:30 smithi163 ceph-mon[27043]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:23:31.135 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:30 smithi163 ceph-mon[27043]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T22:23:31.135 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:31.135 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:31.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:30 smithi081 ceph-mon[24860]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:23:31.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:30 smithi081 ceph-mon[24860]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:23:31.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:30 smithi081 ceph-mon[24860]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T22:23:31.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:31.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:32.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:31 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:23:32.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:31 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:23:32.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:31 smithi163 ceph-mon[27043]: osd.6 [v2:172.21.15.163:6816/2316852861,v1:172.21.15.163:6817/2316852861] boot 2024-04-15T22:23:32.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:31 smithi163 ceph-mon[27043]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T22:23:32.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:31 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:32.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:31 smithi163 ceph-mon[27043]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T22:23:32.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:31 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:23:32.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:31 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:23:32.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:31 smithi081 ceph-mon[24860]: osd.6 [v2:172.21.15.163:6816/2316852861,v1:172.21.15.163:6817/2316852861] boot 2024-04-15T22:23:32.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:31 smithi081 ceph-mon[24860]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T22:23:32.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:31 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:23:32.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:31 smithi081 ceph-mon[24860]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T22:23:32.409 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T22:23:32.410 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T22:23:32.410 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-04-15T22:23:32.410 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-04-15T22:23:32.410 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122776 s, 854 MB/s 2024-04-15T22:23:32.410 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 2024-04-15T22:23:32.410 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-04-15T22:23:33.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_1 2024-04-15T22:23:33.132 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:32 smithi163 ceph-mon[27043]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:33.132 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:32 smithi163 ceph-mon[27043]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T22:23:33.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:32 smithi081 ceph-mon[24860]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:33.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:32 smithi081 ceph-mon[24860]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T22:23:34.591 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T22:23:34.591 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:34.591 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:34.591 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:34.591 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:34.592 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:34.592 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:34.592 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:34.592 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:34.592 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:34.592 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:34.592 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:34.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:34.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:35 smithi081 ceph-mon[24860]: Detected new or changed devices on smithi163 2024-04-15T22:23:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:35 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi163 to 5312M 2024-04-15T22:23:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:35 smithi081 ceph-mon[24860]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T22:23:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:23:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:23:35.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:35 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:35.871 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:35 smithi163 ceph-mon[27043]: Detected new or changed devices on smithi163 2024-04-15T22:23:35.871 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:35 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi163 to 5312M 2024-04-15T22:23:35.871 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:35 smithi163 ceph-mon[27043]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T22:23:35.871 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:23:35.872 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:23:35.872 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:35 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:36.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:36 smithi081 ceph-mon[24860]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:23:36.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:36 smithi081 ceph-mon[24860]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:36.833 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:36 smithi163 ceph-mon[27043]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:23:36.833 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:36 smithi163 ceph-mon[27043]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:38 smithi081 ceph-mon[24860]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-15T22:23:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:38 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/2755766321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b11111-356f-4056-8f67-98dbacc09d52"}]: dispatch 2024-04-15T22:23:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:38 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b11111-356f-4056-8f67-98dbacc09d52"}]: dispatch 2024-04-15T22:23:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:38 smithi081 ceph-mon[24860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80b11111-356f-4056-8f67-98dbacc09d52"}]': finished 2024-04-15T22:23:39.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:38 smithi081 ceph-mon[24860]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T22:23:39.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:39.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:38 smithi081 ceph-mon[24860]: from='client.? 172.21.15.163:0/4136102659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:23:39.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:38 smithi163 ceph-mon[27043]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-15T22:23:39.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:38 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/2755766321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b11111-356f-4056-8f67-98dbacc09d52"}]: dispatch 2024-04-15T22:23:39.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:38 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b11111-356f-4056-8f67-98dbacc09d52"}]: dispatch 2024-04-15T22:23:39.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:38 smithi163 ceph-mon[27043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80b11111-356f-4056-8f67-98dbacc09d52"}]': finished 2024-04-15T22:23:39.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:38 smithi163 ceph-mon[27043]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T22:23:39.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:39.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:38 smithi163 ceph-mon[27043]: from='client.? 172.21.15.163:0/4136102659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:23:40.983 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:40 smithi163 ceph-mon[27043]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T22:23:41.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:40 smithi081 ceph-mon[24860]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T22:23:43.034 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:42 smithi163 ceph-mon[27043]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T22:23:43.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:42 smithi081 ceph-mon[24860]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T22:23:44.783 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:44 smithi163 ceph-mon[27043]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T22:23:44.783 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:44 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T22:23:44.783 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:44 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:44.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:44 smithi081 ceph-mon[24860]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T22:23:44.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:44 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T22:23:44.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:44 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:45.726 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:45 smithi163 ceph-mon[27043]: Deploying daemon osd.7 on smithi163 2024-04-15T22:23:45.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:45 smithi081 ceph-mon[24860]: Deploying daemon osd.7 on smithi163 2024-04-15T22:23:46.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:46 smithi163 ceph-mon[27043]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-15T22:23:46.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:46 smithi081 ceph-mon[24860]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-15T22:23:48.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:48.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:48 smithi081 ceph-mon[24860]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:49.060 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:48 smithi163 ceph-mon[27043]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:50.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:50.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:50.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:50.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:50 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:50.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:50 smithi081 ceph-mon[24860]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:50.625 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:50.625 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:50.625 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:50.625 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:50 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:50.625 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:50 smithi163 ceph-mon[27043]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:52.467 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 7 on host 'smithi163' 2024-04-15T22:23:53.047 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:52 smithi163 ceph-mon[27043]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:53.047 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:52 smithi163 ceph-mon[27043]: from='osd.7 [v2:172.21.15.163:6824/2450203295,v1:172.21.15.163:6825/2450203295]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:23:53.048 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:52 smithi163 ceph-mon[27043]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:23:53.048 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:52 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:53.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:52 smithi081 ceph-mon[24860]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:53.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:52 smithi081 ceph-mon[24860]: from='osd.7 [v2:172.21.15.163:6824/2450203295,v1:172.21.15.163:6825/2450203295]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:23:53.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:52 smithi081 ceph-mon[24860]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:23:53.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:52 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:53.071 DEBUG:teuthology.orchestra.run.smithi163:osd.7> sudo journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.7.service 2024-04-15T22:23:53.073 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-15T22:23:53.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-15T22:23:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:53 smithi081 ceph-mon[24860]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T22:23:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:53 smithi081 ceph-mon[24860]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T22:23:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:53 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:53 smithi081 ceph-mon[24860]: from='osd.7 [v2:172.21.15.163:6824/2450203295,v1:172.21.15.163:6825/2450203295]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:54.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:53 smithi081 ceph-mon[24860]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:54.128 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:53 smithi163 ceph-mon[27043]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T22:23:54.128 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:53 smithi163 ceph-mon[27043]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T22:23:54.128 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:53 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:54.128 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:53 smithi163 ceph-mon[27043]: from='osd.7 [v2:172.21.15.163:6824/2450203295,v1:172.21.15.163:6825/2450203295]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:54.128 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:53 smithi163 ceph-mon[27043]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-04-15T22:23:54.128 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:23:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:23:53.753+0000 7fcb1430d700 -1 osd.7 0 waiting for initial osdmap 2024-04-15T22:23:54.128 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:23:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:23:53.760+0000 7fcb0d4a1700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:23:54.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:54 smithi163 ceph-mon[27043]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:54.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:54 smithi163 ceph-mon[27043]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:23:54.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:54 smithi163 ceph-mon[27043]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T22:23:54.829 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:54 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:54.830 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:54 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:54.830 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:54 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:55.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:54 smithi081 ceph-mon[24860]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:23:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:54 smithi081 ceph-mon[24860]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-04-15T22:23:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:54 smithi081 ceph-mon[24860]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T22:23:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:55.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:54 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:55.121 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:23:55.871 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1713219834,"num_in_osds":8,"osd_in_since":1713219817,"num_remapped_pgs":1} 2024-04-15T22:23:55.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:23:56.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:55 smithi081 ceph-mon[24860]: purged_snaps scrub starts 2024-04-15T22:23:56.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:55 smithi081 ceph-mon[24860]: purged_snaps scrub ok 2024-04-15T22:23:56.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:55 smithi081 ceph-mon[24860]: osd.7 [v2:172.21.15.163:6824/2450203295,v1:172.21.15.163:6825/2450203295] boot 2024-04-15T22:23:56.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:55 smithi081 ceph-mon[24860]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T22:23:56.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:55 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:56.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:55 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3131388358' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T22:23:56.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:55 smithi081 ceph-mon[24860]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T22:23:56.091 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:23:56.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:55 smithi163 ceph-mon[27043]: purged_snaps scrub starts 2024-04-15T22:23:56.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:55 smithi163 ceph-mon[27043]: purged_snaps scrub ok 2024-04-15T22:23:56.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:55 smithi163 ceph-mon[27043]: osd.7 [v2:172.21.15.163:6824/2450203295,v1:172.21.15.163:6825/2450203295] boot 2024-04-15T22:23:56.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:55 smithi163 ceph-mon[27043]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T22:23:56.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:55 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:23:56.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:55 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/3131388358' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T22:23:56.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:55 smithi163 ceph-mon[27043]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T22:23:56.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:56 smithi081 ceph-mon[24860]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:56.805 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:56 smithi081 ceph-mon[24860]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T22:23:56.855 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:56 smithi163 ceph-mon[27043]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:56.856 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:56 smithi163 ceph-mon[27043]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T22:23:57.802 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:23:57.802 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":58,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","created":"2024-04-15T22:17:50.268370+0000","modified":"2024-04-15T22:23:57.759912+0000","last_up_change":"2024-04-15T22:23:54.747071+0000","last_in_change":"2024-04-15T22:23:37.855814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:21:55.965072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4a88478d-6021-4cd6-89d7-6439d2cb93fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6803","nonce":3069676866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6805","nonce":3069676866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6809","nonce":3069676866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6807","nonce":3069676866}]},"public_addr":"172.21.15.81:6803/3069676866","cluster_addr":"172.21.15.81:6805/3069676866","heartbeat_back_addr":"172.21.15.81:6809/3069676866","heartbeat_front_addr":"172.21.15.81:6807/3069676866","state":["exists","up"]},{"osd":1,"uuid":"40fc7f9b-5bec-4774-ba1e-f5be50b9f05c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6811","nonce":2281467360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6813","nonce":2281467360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6817","nonce":2281467360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6815","nonce":2281467360}]},"public_addr":"172.21.15.81:6811/2281467360","cluster_addr":"172.21.15.81:6813/2281467360","heartbeat_back_addr":"172.21.15.81:6817/2281467360","heartbeat_front_addr":"172.21.15.81:6815/2281467360","state":["exists","up"]},{"osd":2,"uuid":"f0091596-5d84-461a-ba14-6c900cc9af14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6819","nonce":2961844695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6821","nonce":2961844695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6825","nonce":2961844695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6823","nonce":2961844695}]},"public_addr":"172.21.15.81:6819/2961844695","cluster_addr":"172.21.15.81:6821/2961844695","heartbeat_back_addr":"172.21.15.81:6825/2961844695","heartbeat_front_addr":"172.21.15.81:6823/2961844695","state":["exists","up"]},{"osd":3,"uuid":"38f7a45b-9e4a-4a7f-8a58-18b0818db1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6827","nonce":3454580015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6829","nonce":3454580015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6833","nonce":3454580015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6831","nonce":3454580015}]},"public_addr":"172.21.15.81:6827/3454580015","cluster_addr":"172.21.15.81:6829/3454580015","heartbeat_back_addr":"172.21.15.81:6833/3454580015","heartbeat_front_addr":"172.21.15.81:6831/3454580015","state":["exists","up"]},{"osd":4,"uuid":"801b8f37-6b69-4960-9f37-caecbf0f6113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6801","nonce":902397810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6803","nonce":902397810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6807","nonce":902397810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6805","nonce":902397810}]},"public_addr":"172.21.15.163:6801/902397810","cluster_addr":"172.21.15.163:6803/902397810","heartbeat_back_addr":"172.21.15.163:6807/902397810","heartbeat_front_addr":"172.21.15.163:6805/902397810","state":["exists","up"]},{"osd":5,"uuid":"2727f5ae-a8e6-4677-a2a4-74c3c9810a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6809","nonce":4193470634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6811","nonce":4193470634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6815","nonce":4193470634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6813","nonce":4193470634}]},"public_addr":"172.21.15.163:6809/4193470634","cluster_addr":"172.21.15.163:6811/4193470634","heartbeat_back_addr":"172.21.15.163:6815/4193470634","heartbeat_front_addr":"172.21.15.163:6813/4193470634","state":["exists","up"]},{"osd":6,"uuid":"bd789380-c4f0-4eef-a592-d3819599d2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6817","nonce":2316852861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6819","nonce":2316852861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6823","nonce":2316852861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6821","nonce":2316852861}]},"public_addr":"172.21.15.163:6817/2316852861","cluster_addr":"172.21.15.163:6819/2316852861","heartbeat_back_addr":"172.21.15.163:6823/2316852861","heartbeat_front_addr":"172.21.15.163:6821/2316852861","state":["exists","up"]},{"osd":7,"uuid":"80b11111-356f-4056-8f67-98dbacc09d52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6825","nonce":2450203295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6827","nonce":2450203295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6831","nonce":2450203295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6829","nonce":2450203295}]},"public_addr":"172.21.15.163:6825/2450203295","cluster_addr":"172.21.15.163:6827/2450203295","heartbeat_back_addr":"172.21.15.163:6831/2450203295","heartbeat_front_addr":"172.21.15.163:6829/2450203295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:53.386581+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:25.688824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:53.672674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:19.000342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:41.335552+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:05.149176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:29.018059+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:53.334527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/3363741099":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/4188390236":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/1185517108":"2024-04-16T22:19:46.706744+0000","172.21.15.81:6800/2486074512":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/2748180865":"2024-04-16T22:18:53.798820+0000","172.21.15.81:6801/3884072308":"2024-04-16T22:18:53.798820+0000","172.21.15.81:6800/3884072308":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/1964432423":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/3175879974":"2024-04-16T22:18:24.227835+0000","172.21.15.81:0/3799054035":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/1763660374":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6801/3535508663":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6801/2486074512":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/2089157550":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6800/3535508663":"2024-04-16T22:18:24.227835+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:23:58.493 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: Detected new or changed devices on smithi163 2024-04-15T22:23:58.493 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.493 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: Adjusting osd_memory_target on smithi163 to 3984M 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T22:23:58.494 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:58 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2221808191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:58.518 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T22:21:55.965072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-15T22:23:58.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-15T22:23:58.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: Detected new or changed devices on smithi163 2024-04-15T22:23:58.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: Adjusting osd_memory_target on smithi163 to 3984M 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T22:23:58.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:58 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2221808191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:58.741 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:23:59.766 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:23:59 smithi081 ceph-mon[24860]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T22:24:00.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:23:59 smithi163 ceph-mon[27043]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T22:24:00.406 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-15T22:24:01.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:00 smithi081 ceph-mon[24860]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:01.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:00 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2941358699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:24:01.165 INFO:tasks.cephadm:Setting up client nodes... 2024-04-15T22:24:01.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T22:24:01.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:00 smithi163 ceph-mon[27043]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:01.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:00 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2941358699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:24:03.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:02 smithi081 ceph-mon[24860]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-15T22:24:03.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:02 smithi163 ceph-mon[27043]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-15T22:24:03.272 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-04-15T22:24:03.272 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQADqR1mHJwYEBAA0Pibr7RxCRluTa1OEFGFuQ== 2024-04-15T22:24:03.964 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:03 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1388337123' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:24:03.964 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:03 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1388337123' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:24:03.988 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-15T22:24:03.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-15T22:24:03.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T22:24:04.031 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T22:24:04.031 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-15T22:24:04.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-15T22:24:04.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:03 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1388337123' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:24:04.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:03 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1388337123' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:24:04.286 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:05.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:04 smithi081 ceph-mon[24860]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-15T22:24:05.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:04 smithi163 ceph-mon[27043]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-15T22:24:06.192 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:24:06.845 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":19,"active_gid":14204,"active_name":"smithi081.wfvsja","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":98336810},{"type":"v1","addr":"172.21.15.81:6801","nonce":98336810}]},"active_addr":"172.21.15.81:6801/98336810","active_change":"2024-04-15T22:19:46.707058+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14224,"name":"smithi163.cdbjip","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":321515121}]},{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2941094782}]},{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2142777156}]},{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1751779019}]}]}} 2024-04-15T22:24:06.852 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-15T22:24:06.852 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T22:24:06.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:24:07.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:06 smithi081 ceph-mon[24860]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T22:24:07.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:06 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3390513796' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T22:24:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:07.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:06 smithi163 ceph-mon[27043]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T22:24:07.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:06 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/3390513796' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T22:24:08.835 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:24:08.836 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","created":"2024-04-15T22:17:50.268370+0000","modified":"2024-04-15T22:23:58.761108+0000","last_up_change":"2024-04-15T22:23:54.747071+0000","last_in_change":"2024-04-15T22:23:37.855814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:21:55.965072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4a88478d-6021-4cd6-89d7-6439d2cb93fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6803","nonce":3069676866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6805","nonce":3069676866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6809","nonce":3069676866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6807","nonce":3069676866}]},"public_addr":"172.21.15.81:6803/3069676866","cluster_addr":"172.21.15.81:6805/3069676866","heartbeat_back_addr":"172.21.15.81:6809/3069676866","heartbeat_front_addr":"172.21.15.81:6807/3069676866","state":["exists","up"]},{"osd":1,"uuid":"40fc7f9b-5bec-4774-ba1e-f5be50b9f05c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6811","nonce":2281467360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6813","nonce":2281467360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6817","nonce":2281467360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6815","nonce":2281467360}]},"public_addr":"172.21.15.81:6811/2281467360","cluster_addr":"172.21.15.81:6813/2281467360","heartbeat_back_addr":"172.21.15.81:6817/2281467360","heartbeat_front_addr":"172.21.15.81:6815/2281467360","state":["exists","up"]},{"osd":2,"uuid":"f0091596-5d84-461a-ba14-6c900cc9af14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6819","nonce":2961844695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6821","nonce":2961844695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6825","nonce":2961844695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6823","nonce":2961844695}]},"public_addr":"172.21.15.81:6819/2961844695","cluster_addr":"172.21.15.81:6821/2961844695","heartbeat_back_addr":"172.21.15.81:6825/2961844695","heartbeat_front_addr":"172.21.15.81:6823/2961844695","state":["exists","up"]},{"osd":3,"uuid":"38f7a45b-9e4a-4a7f-8a58-18b0818db1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6827","nonce":3454580015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6829","nonce":3454580015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6833","nonce":3454580015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6831","nonce":3454580015}]},"public_addr":"172.21.15.81:6827/3454580015","cluster_addr":"172.21.15.81:6829/3454580015","heartbeat_back_addr":"172.21.15.81:6833/3454580015","heartbeat_front_addr":"172.21.15.81:6831/3454580015","state":["exists","up"]},{"osd":4,"uuid":"801b8f37-6b69-4960-9f37-caecbf0f6113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6801","nonce":902397810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6803","nonce":902397810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6807","nonce":902397810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6805","nonce":902397810}]},"public_addr":"172.21.15.163:6801/902397810","cluster_addr":"172.21.15.163:6803/902397810","heartbeat_back_addr":"172.21.15.163:6807/902397810","heartbeat_front_addr":"172.21.15.163:6805/902397810","state":["exists","up"]},{"osd":5,"uuid":"2727f5ae-a8e6-4677-a2a4-74c3c9810a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6809","nonce":4193470634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6811","nonce":4193470634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6815","nonce":4193470634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6813","nonce":4193470634}]},"public_addr":"172.21.15.163:6809/4193470634","cluster_addr":"172.21.15.163:6811/4193470634","heartbeat_back_addr":"172.21.15.163:6815/4193470634","heartbeat_front_addr":"172.21.15.163:6813/4193470634","state":["exists","up"]},{"osd":6,"uuid":"bd789380-c4f0-4eef-a592-d3819599d2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6817","nonce":2316852861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6819","nonce":2316852861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6823","nonce":2316852861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6821","nonce":2316852861}]},"public_addr":"172.21.15.163:6817/2316852861","cluster_addr":"172.21.15.163:6819/2316852861","heartbeat_back_addr":"172.21.15.163:6823/2316852861","heartbeat_front_addr":"172.21.15.163:6821/2316852861","state":["exists","up"]},{"osd":7,"uuid":"80b11111-356f-4056-8f67-98dbacc09d52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6825","nonce":2450203295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6827","nonce":2450203295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6831","nonce":2450203295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6829","nonce":2450203295}]},"public_addr":"172.21.15.163:6825/2450203295","cluster_addr":"172.21.15.163:6827/2450203295","heartbeat_back_addr":"172.21.15.163:6831/2450203295","heartbeat_front_addr":"172.21.15.163:6829/2450203295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:53.386581+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:25.688824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:53.672674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:19.000342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:41.335552+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:05.149176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:29.018059+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:53.334527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/3363741099":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/4188390236":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/1185517108":"2024-04-16T22:19:46.706744+0000","172.21.15.81:6800/2486074512":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/2748180865":"2024-04-16T22:18:53.798820+0000","172.21.15.81:6801/3884072308":"2024-04-16T22:18:53.798820+0000","172.21.15.81:6800/3884072308":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/1964432423":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/3175879974":"2024-04-16T22:18:24.227835+0000","172.21.15.81:0/3799054035":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/1763660374":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6801/3535508663":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6801/2486074512":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/2089157550":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6800/3535508663":"2024-04-16T22:18:24.227835+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:24:09.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:08 smithi081 ceph-mon[24860]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-15T22:24:09.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:08 smithi163 ceph-mon[27043]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-15T22:24:09.545 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T22:24:09.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:24:09.764 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:10.018 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:09 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/290789015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:24:10.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:09 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/290789015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:24:10.926 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:10 smithi081 ceph-mon[24860]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-04-15T22:24:11.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:10 smithi163 ceph-mon[27043]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-04-15T22:24:11.650 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:24:11.651 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"fsid":"e7d16278-fb75-11ee-bc8f-c7b262605968","created":"2024-04-15T22:17:50.268370+0000","modified":"2024-04-15T22:23:58.761108+0000","last_up_change":"2024-04-15T22:23:54.747071+0000","last_in_change":"2024-04-15T22:23:37.855814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:21:55.965072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4a88478d-6021-4cd6-89d7-6439d2cb93fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6803","nonce":3069676866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6805","nonce":3069676866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6809","nonce":3069676866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3069676866},{"type":"v1","addr":"172.21.15.81:6807","nonce":3069676866}]},"public_addr":"172.21.15.81:6803/3069676866","cluster_addr":"172.21.15.81:6805/3069676866","heartbeat_back_addr":"172.21.15.81:6809/3069676866","heartbeat_front_addr":"172.21.15.81:6807/3069676866","state":["exists","up"]},{"osd":1,"uuid":"40fc7f9b-5bec-4774-ba1e-f5be50b9f05c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6811","nonce":2281467360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6813","nonce":2281467360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6817","nonce":2281467360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2281467360},{"type":"v1","addr":"172.21.15.81:6815","nonce":2281467360}]},"public_addr":"172.21.15.81:6811/2281467360","cluster_addr":"172.21.15.81:6813/2281467360","heartbeat_back_addr":"172.21.15.81:6817/2281467360","heartbeat_front_addr":"172.21.15.81:6815/2281467360","state":["exists","up"]},{"osd":2,"uuid":"f0091596-5d84-461a-ba14-6c900cc9af14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6819","nonce":2961844695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6821","nonce":2961844695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6825","nonce":2961844695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2961844695},{"type":"v1","addr":"172.21.15.81:6823","nonce":2961844695}]},"public_addr":"172.21.15.81:6819/2961844695","cluster_addr":"172.21.15.81:6821/2961844695","heartbeat_back_addr":"172.21.15.81:6825/2961844695","heartbeat_front_addr":"172.21.15.81:6823/2961844695","state":["exists","up"]},{"osd":3,"uuid":"38f7a45b-9e4a-4a7f-8a58-18b0818db1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6827","nonce":3454580015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6829","nonce":3454580015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6833","nonce":3454580015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3454580015},{"type":"v1","addr":"172.21.15.81:6831","nonce":3454580015}]},"public_addr":"172.21.15.81:6827/3454580015","cluster_addr":"172.21.15.81:6829/3454580015","heartbeat_back_addr":"172.21.15.81:6833/3454580015","heartbeat_front_addr":"172.21.15.81:6831/3454580015","state":["exists","up"]},{"osd":4,"uuid":"801b8f37-6b69-4960-9f37-caecbf0f6113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6801","nonce":902397810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6803","nonce":902397810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6807","nonce":902397810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":902397810},{"type":"v1","addr":"172.21.15.163:6805","nonce":902397810}]},"public_addr":"172.21.15.163:6801/902397810","cluster_addr":"172.21.15.163:6803/902397810","heartbeat_back_addr":"172.21.15.163:6807/902397810","heartbeat_front_addr":"172.21.15.163:6805/902397810","state":["exists","up"]},{"osd":5,"uuid":"2727f5ae-a8e6-4677-a2a4-74c3c9810a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6809","nonce":4193470634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6811","nonce":4193470634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6815","nonce":4193470634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4193470634},{"type":"v1","addr":"172.21.15.163:6813","nonce":4193470634}]},"public_addr":"172.21.15.163:6809/4193470634","cluster_addr":"172.21.15.163:6811/4193470634","heartbeat_back_addr":"172.21.15.163:6815/4193470634","heartbeat_front_addr":"172.21.15.163:6813/4193470634","state":["exists","up"]},{"osd":6,"uuid":"bd789380-c4f0-4eef-a592-d3819599d2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6817","nonce":2316852861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6819","nonce":2316852861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6823","nonce":2316852861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2316852861},{"type":"v1","addr":"172.21.15.163:6821","nonce":2316852861}]},"public_addr":"172.21.15.163:6817/2316852861","cluster_addr":"172.21.15.163:6819/2316852861","heartbeat_back_addr":"172.21.15.163:6823/2316852861","heartbeat_front_addr":"172.21.15.163:6821/2316852861","state":["exists","up"]},{"osd":7,"uuid":"80b11111-356f-4056-8f67-98dbacc09d52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6825","nonce":2450203295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6827","nonce":2450203295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6831","nonce":2450203295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2450203295},{"type":"v1","addr":"172.21.15.163:6829","nonce":2450203295}]},"public_addr":"172.21.15.163:6825/2450203295","cluster_addr":"172.21.15.163:6827/2450203295","heartbeat_back_addr":"172.21.15.163:6831/2450203295","heartbeat_front_addr":"172.21.15.163:6829/2450203295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:53.386581+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:25.688824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:53.672674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:19.000342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:41.335552+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:05.149176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:29.018059+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:23:53.334527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/3363741099":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/4188390236":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/1185517108":"2024-04-16T22:19:46.706744+0000","172.21.15.81:6800/2486074512":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/2748180865":"2024-04-16T22:18:53.798820+0000","172.21.15.81:6801/3884072308":"2024-04-16T22:18:53.798820+0000","172.21.15.81:6800/3884072308":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/1964432423":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/3175879974":"2024-04-16T22:18:24.227835+0000","172.21.15.81:0/3799054035":"2024-04-16T22:18:53.798820+0000","172.21.15.81:0/1763660374":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6801/3535508663":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6801/2486074512":"2024-04-16T22:19:46.706744+0000","172.21.15.81:0/2089157550":"2024-04-16T22:18:24.227835+0000","172.21.15.81:6800/3535508663":"2024-04-16T22:18:24.227835+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:24:12.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:11 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2805174701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:24:12.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:11 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2805174701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:24:12.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-15T22:24:12.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-15T22:24:12.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-15T22:24:12.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-15T22:24:12.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-15T22:24:12.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-15T22:24:12.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-15T22:24:12.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-15T22:24:12.792 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:12 smithi081 ceph-mon[24860]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-15T22:24:12.799 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:12.799 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:12.800 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:12.800 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:12.807 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:13.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:12 smithi163 ceph-mon[27043]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-15T22:24:14.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:14 smithi081 ceph-mon[24860]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:14.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:14 smithi163 ceph-mon[27043]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:17.019 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:16 smithi081 ceph-mon[24860]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:17.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:16 smithi163 ceph-mon[27043]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:19.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:18 smithi081 ceph-mon[24860]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:19.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:18 smithi163 ceph-mon[27043]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:20.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:20 smithi081 ceph-mon[24860]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:21.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:20 smithi163 ceph-mon[27043]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:23.005 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:22 smithi081 ceph-mon[24860]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:23.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:22 smithi163 ceph-mon[27043]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:24.781 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:24 smithi081 ceph-mon[24860]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:24.805 INFO:teuthology.orchestra.run.smithi081.stdout:38654705709 2024-04-15T22:24:24.805 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-15T22:24:24.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:24 smithi163 ceph-mon[27043]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:27.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:26 smithi081 ceph-mon[24860]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:27.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:26 smithi163 ceph-mon[27043]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:27.377 INFO:teuthology.orchestra.run.smithi081.stdout:167503724563 2024-04-15T22:24:27.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-15T22:24:27.863 INFO:teuthology.orchestra.run.smithi081.stdout:201863462925 2024-04-15T22:24:27.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-15T22:24:28.274 INFO:teuthology.orchestra.run.smithi081.stdout:60129542183 2024-04-15T22:24:28.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-15T22:24:28.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:28 smithi081 ceph-mon[24860]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:29.037 INFO:teuthology.orchestra.run.smithi081.stdout:133143986199 2024-04-15T22:24:29.037 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-15T22:24:29.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:28 smithi163 ceph-mon[27043]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:29.651 INFO:teuthology.orchestra.run.smithi081.stdout:111669149725 2024-04-15T22:24:29.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-15T22:24:29.868 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:30.252 INFO:teuthology.orchestra.run.smithi081.stdout:236223201289 2024-04-15T22:24:30.252 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-15T22:24:30.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:30 smithi081 ceph-mon[24860]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:30.893 INFO:teuthology.orchestra.run.smithi081.stdout:81604378655 2024-04-15T22:24:30.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-15T22:24:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:30 smithi163 ceph-mon[27043]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:31.461 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:31.461 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:31.461 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:31.565 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:32.401 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:32.401 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:32.912 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:32 smithi081 ceph-mon[24860]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:33.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:32 smithi163 ceph-mon[27043]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:33.934 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:34.804 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:34 smithi081 ceph-mon[24860]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:34.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:34 smithi163 ceph-mon[27043]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:36.165 INFO:teuthology.orchestra.run.smithi081.stdout:38654705712 2024-04-15T22:24:36.336 INFO:teuthology.orchestra.run.smithi081.stdout:167503724565 2024-04-15T22:24:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:36 smithi081 ceph-mon[24860]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:36 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3582054634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T22:24:37.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:36 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2513964194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T22:24:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:36 smithi163 ceph-mon[27043]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:36 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/3582054634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T22:24:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:36 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2513964194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T22:24:37.327 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705712 for osd.0 2024-04-15T22:24:37.327 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:39.006 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:38 smithi081 ceph-mon[24860]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:39.030 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724563 got 167503724565 for osd.5 2024-04-15T22:24:39.030 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:38 smithi163 ceph-mon[27043]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:40.330 INFO:teuthology.orchestra.run.smithi081.stdout:201863462928 2024-04-15T22:24:40.986 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:40 smithi081 ceph-mon[24860]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:40.986 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:40 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2800816685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T22:24:41.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:40 smithi163 ceph-mon[27043]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:41.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:40 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2800816685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T22:24:42.393 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462925 got 201863462928 for osd.6 2024-04-15T22:24:42.393 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:42.762 INFO:teuthology.orchestra.run.smithi081.stdout:111669149727 2024-04-15T22:24:43.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:42 smithi081 ceph-mon[24860]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:43.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:42 smithi163 ceph-mon[27043]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:43.601 INFO:teuthology.orchestra.run.smithi081.stdout:81604378660 2024-04-15T22:24:44.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:43 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/774897379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T22:24:44.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:43 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/385954593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T22:24:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:43 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/774897379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T22:24:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:43 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/385954593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T22:24:44.637 INFO:teuthology.orchestra.run.smithi081.stdout:133143986203 2024-04-15T22:24:44.853 INFO:teuthology.orchestra.run.smithi081.stdout:236223201292 2024-04-15T22:24:45.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:44 smithi081 ceph-mon[24860]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:45.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:44 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2480017774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T22:24:45.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:44 smithi163 ceph-mon[27043]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:45.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:44 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2480017774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T22:24:45.439 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201289 got 236223201292 for osd.7 2024-04-15T22:24:45.439 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:45.903 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149725 got 111669149727 for osd.3 2024-04-15T22:24:45.904 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:46.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:45 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2395419628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T22:24:46.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:45 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2395419628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T22:24:46.273 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378655 got 81604378660 for osd.2 2024-04-15T22:24:46.274 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:46.742 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986199 got 133143986203 for osd.4 2024-04-15T22:24:46.743 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:47.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:46 smithi081 ceph-mon[24860]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:47.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:46 smithi163 ceph-mon[27043]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:47.469 INFO:teuthology.orchestra.run.smithi081.stdout:60129542187 2024-04-15T22:24:48.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:47 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/4188440247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T22:24:48.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:48.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:48.128 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542187 for osd.1 2024-04-15T22:24:48.128 DEBUG:teuthology.parallel:result is None 2024-04-15T22:24:48.128 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T22:24:48.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T22:24:48.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:47 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/4188440247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T22:24:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:48.350 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:48 smithi081 ceph-mon[24860]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:48 smithi163 ceph-mon[27043]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:50.209 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:24:50.211 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-04-15T22:24:50.859 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-15T22:24:49.697677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48924,"kb_used_data":6300,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912292,"statfs":{"total":767960285184,"available":767910187008,"internally_reserved":0,"allocated":6451200,"data_stored":3546677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002003"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-15T22:23:58.767150+0000","last_change":"2024-04-15T22:23:58.767150+0000","last_active":"2024-04-15T22:23:58.767150+0000","last_peered":"2024-04-15T22:23:58.767150+0000","last_clean":"2024-04-15T22:23:58.767150+0000","last_became_active":"2024-04-15T22:23:58.766591+0000","last_became_peered":"2024-04-15T22:23:58.766591+0000","last_unstale":"2024-04-15T22:23:58.767150+0000","last_undegraded":"2024-04-15T22:23:58.767150+0000","last_fullsized":"2024-04-15T22:23:58.767150+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T22:21:55.965440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T22:21:55.965440+0000","last_clean_scrub_stamp":"2024-04-15T22:21:55.965440+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:26:01.611250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.5}]},{"osd":4,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.34599999999999997,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.32400000000000001},"min":{"1min":0.23899999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Apr 15 22:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.375,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.35999999999999999},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.41399999999999998,"15min":0.40899999999999997},"min":{"1min":0.318,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46000000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.56399999999999995},"min":{"1min":0.496,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.75700000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.43099999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Apr 15 22:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6948,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738204,"statfs":{"total":95995035648,"available":95987920896,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 22:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.30399999999999999},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.36899999999999999,"15min":0.33700000000000002},"min":{"1min":0.26700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Apr 15 22:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.32400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63700000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.33000000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.29299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.502},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.33600000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.45900000000000002}]},{"osd":4,"last update":"Mon Apr 15 22:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Apr 15 22:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.60299999999999998}]},{"osd":6,"last update":"Mon Apr 15 22:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378661,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38100000000000001,"15min":0.38300000000000001},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.35599999999999998,"15min":0.34799999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.38400000000000001,"15min":0.376},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.37},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.40699999999999997,"15min":0.40400000000000003},"min":{"1min":0.313,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.33900000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42699999999999999,"15min":0.42899999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59499999999999997,"5min":0.63,"15min":0.63},"last":0.44},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.438,"15min":0.442},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.58299999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Apr 15 22:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.499},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Apr 15 22:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38600000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58299999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.38800000000000001},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.53500000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.33200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63600000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.438,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61899999999999999,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72399999999999998,"5min":0.752,"15min":0.752},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Apr 15 22:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41799999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.629}]},{"osd":1,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.5},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.47499999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.45200000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.432}]},{"osd":6,"last update":"Mon Apr 15 22:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.47899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.37},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.443},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.501},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T22:24:50.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T22:24:51.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:50 smithi081 ceph-mon[24860]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:51.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:50 smithi163 ceph-mon[27043]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:52.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:51 smithi081 ceph-mon[24860]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:24:52.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:51 smithi163 ceph-mon[27043]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:24:52.840 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:24:52.842 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-04-15T22:24:53.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:52 smithi081 ceph-mon[24860]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:53.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:52 smithi163 ceph-mon[27043]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:53.567 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-04-15T22:24:51.698142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48924,"kb_used_data":6300,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912292,"statfs":{"total":767960285184,"available":767910187008,"internally_reserved":0,"allocated":6451200,"data_stored":3546677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002201"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-15T22:23:58.767150+0000","last_change":"2024-04-15T22:23:58.767150+0000","last_active":"2024-04-15T22:23:58.767150+0000","last_peered":"2024-04-15T22:23:58.767150+0000","last_clean":"2024-04-15T22:23:58.767150+0000","last_became_active":"2024-04-15T22:23:58.766591+0000","last_became_peered":"2024-04-15T22:23:58.766591+0000","last_unstale":"2024-04-15T22:23:58.767150+0000","last_undegraded":"2024-04-15T22:23:58.767150+0000","last_fullsized":"2024-04-15T22:23:58.767150+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T22:21:55.965440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T22:21:55.965440+0000","last_clean_scrub_stamp":"2024-04-15T22:21:55.965440+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:26:01.611250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.496},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.628}]},{"osd":2,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59299999999999997}]},{"osd":5,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.70099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.34599999999999997,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.32400000000000001},"min":{"1min":0.23899999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Apr 15 22:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.375,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.35999999999999999},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.41399999999999998,"15min":0.40899999999999997},"min":{"1min":0.318,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46000000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.56399999999999995},"min":{"1min":0.496,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.75700000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.43099999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Apr 15 22:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6948,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738204,"statfs":{"total":95995035648,"available":95987920896,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 22:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.30399999999999999},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.36899999999999999,"15min":0.33700000000000002},"min":{"1min":0.26700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.31}]},{"osd":2,"last update":"Mon Apr 15 22:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.32400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63700000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.33000000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.29299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.33600000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.48899999999999999}]},{"osd":4,"last update":"Mon Apr 15 22:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.438,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.438}]},{"osd":5,"last update":"Mon Apr 15 22:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.501},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Apr 15 22:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38100000000000001,"15min":0.38300000000000001},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.35599999999999998,"15min":0.34799999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.38400000000000001,"15min":0.376},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.37},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.40699999999999997,"15min":0.40400000000000003},"min":{"1min":0.313,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.33900000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42699999999999999,"15min":0.42899999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59499999999999997,"5min":0.63,"15min":0.63},"last":0.44},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.438,"15min":0.442},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.58299999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Apr 15 22:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.499},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Apr 15 22:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38600000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58299999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.38800000000000001},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.53500000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Apr 15 22:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.33200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63600000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.438,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61899999999999999,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72399999999999998,"5min":0.752,"15min":0.752},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Apr 15 22:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41799999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.629}]},{"osd":1,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.5},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.47499999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.45200000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.432}]},{"osd":6,"last update":"Mon Apr 15 22:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.47899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.37},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.443},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Apr 15 22:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.501},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Apr 15 22:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T22:24:53.568 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T22:24:53.568 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T22:24:53.568 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-15T22:24:53.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-15T22:24:53.793 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/mon.smithi081/config 2024-04-15T22:24:53.816 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:53 smithi081 ceph-mon[24860]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:24:54.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:53 smithi163 ceph-mon[27043]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:24:54.896 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:54 smithi081 ceph-mon[24860]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:55.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:54 smithi163 ceph-mon[27043]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:55.594 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:24:55.595 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T22:24:56.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:55 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2217555684' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T22:24:56.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:55 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2217555684' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T22:24:56.284 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-15T22:24:56.284 INFO:tasks.cephadm:Setup complete, yielding 2024-04-15T22:24:56.285 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:24:56.295 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:24:56.295 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch status' 2024-04-15T22:24:57.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:56 smithi081 ceph-mon[24860]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:57.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:56 smithi163 ceph-mon[27043]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:57.879 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:57.879 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:57.879 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:57.879 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:57 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:24:57.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:57.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:57.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:57.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:57 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:24:58.308 INFO:teuthology.orchestra.run.smithi081.stdout:Backend: cephadm 2024-04-15T22:24:58.308 INFO:teuthology.orchestra.run.smithi081.stdout:Available: Yes 2024-04-15T22:24:58.308 INFO:teuthology.orchestra.run.smithi081.stdout:Paused: No 2024-04-15T22:24:58.967 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:58 smithi081 ceph-mon[24860]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:58.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-15T22:24:59.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:58 smithi163 ceph-mon[27043]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:00.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:24:59 smithi081 ceph-mon[24860]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:00.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:24:59 smithi163 ceph-mon[27043]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:alertmanager.smithi081 smithi081 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 41284a394202 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:crash.smithi081 smithi081 running (5m) 2m ago 5m 6400k - 17.2.0 e1d6a67b021e 543e2e5d66e2 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:crash.smithi163 smithi163 running (5m) 66s ago 5m 6400k - 17.2.0 e1d6a67b021e e2865e86353b 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:grafana.smithi081 smithi081 *:3000 running (4m) 2m ago 5m 45.3M - 8.3.5 dad864ee21e9 a95ecd98442c 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.smithi081.wfvsja smithi081 *:9283 running (6m) 2m ago 6m 447M - 17.2.0 e1d6a67b021e f996fd46940d 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.smithi163.cdbjip smithi163 *:8443,9283 running (4m) 66s ago 4m 411M - 17.2.0 e1d6a67b021e 37f9a7722efd 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:mon.smithi081 smithi081 running (6m) 2m ago 7m 41.3M 2048M 17.2.0 e1d6a67b021e 8622084b079c 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:mon.smithi163 smithi163 running (4m) 66s ago 4m 40.0M 2048M 17.2.0 e1d6a67b021e 06c7e5f8d2ba 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:node-exporter.smithi081 smithi081 *:9100 running (5m) 2m ago 5m 17.9M - 1dbe0e931976 6205702b4b03 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:node-exporter.smithi163 smithi163 *:9100 running (4m) 66s ago 4m 16.9M - 1dbe0e931976 7c2719210d6d 2024-04-15T22:25:00.945 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 smithi081 running (4m) 2m ago 4m 35.2M 3216M 17.2.0 e1d6a67b021e b7b6a575e49b 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 smithi081 running (3m) 2m ago 3m 37.2M 3216M 17.2.0 e1d6a67b021e c8f49993a72b 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 smithi081 running (3m) 2m ago 3m 30.9M 3216M 17.2.0 e1d6a67b021e cfba92df08d2 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 smithi081 running (2m) 2m ago 2m 11.0M 3216M 17.2.0 e1d6a67b021e d878f89983b5 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 smithi163 running (2m) 66s ago 2m 34.2M 3984M 17.2.0 e1d6a67b021e 4e994b835b7e 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 smithi163 running (2m) 66s ago 2m 32.9M 3984M 17.2.0 e1d6a67b021e a25f44d5b877 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 smithi163 running (96s) 66s ago 96s 31.2M 3984M 17.2.0 e1d6a67b021e c695e1b9459d 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 smithi163 running (72s) 66s ago 71s 10.7M 3984M 17.2.0 e1d6a67b021e 8da0636b871f 2024-04-15T22:25:00.946 INFO:teuthology.orchestra.run.smithi081.stdout:prometheus.smithi081 smithi081 *:9095 running (4m) 2m ago 4m 49.0M - 514e6a882f6e 577580326da4 2024-04-15T22:25:01.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:00 smithi163 ceph-mon[27043]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:01.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:00 smithi081 ceph-mon[24860]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:01.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-15T22:25:01.931 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:01 smithi081 ceph-mon[24860]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:02.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:01 smithi163 ceph-mon[27043]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:02.944 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:02 smithi081 ceph-mon[24860]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:03.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:02 smithi163 ceph-mon[27043]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:03.644 INFO:teuthology.orchestra.run.smithi081.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:crash 2/2 2m ago 6m * 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:mgr 2/2 2m ago 6m count:2 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:mon 2/2 2m ago 5m smithi081:172.21.15.81=smithi081;smithi163:172.21.15.163=smithi163;count:2 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:osd 8 2m ago - 2024-04-15T22:25:03.645 INFO:teuthology.orchestra.run.smithi081.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-15T22:25:04.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-15T22:25:04.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:04 smithi081 ceph-mon[24860]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:04.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:04 smithi081 ceph-mon[24860]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:04.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:04 smithi163 ceph-mon[27043]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:04.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:04 smithi163 ceph-mon[27043]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:06.266 INFO:teuthology.orchestra.run.smithi081.stdout:HOST ADDR LABELS STATUS 2024-04-15T22:25:06.267 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 172.21.15.81 2024-04-15T22:25:06.267 INFO:teuthology.orchestra.run.smithi081.stdout:smithi163 172.21.15.163 2024-04-15T22:25:06.267 INFO:teuthology.orchestra.run.smithi081.stdout:2 hosts in cluster 2024-04-15T22:25:07.031 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:06 smithi081 ceph-mon[24860]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:07.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-15T22:25:07.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:06 smithi163 ceph-mon[27043]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:08.021 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:07 smithi081 ceph-mon[24860]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:08.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:07 smithi163 ceph-mon[27043]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:09.041 INFO:teuthology.orchestra.run.smithi081.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-15T22:25:09.041 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400N7400BGN 400G LVM detected, locked 2024-04-15T22:25:09.041 INFO:teuthology.orchestra.run.smithi081.stdout:smithi163 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GU400BGN 400G LVM detected, locked 2024-04-15T22:25:09.042 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:08 smithi081 ceph-mon[24860]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:09.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:08 smithi163 ceph-mon[27043]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:09.671 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-15T22:25:09.680 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:25:09.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-15T22:25:09.712 INFO:teuthology.orchestra.run.smithi081.stderr:+ systemctl stop nfs-server 2024-04-15T22:25:09.720 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi163.front.sepia.ceph.com 2024-04-15T22:25:09.720 DEBUG:teuthology.orchestra.run.smithi163:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-15T22:25:09.755 INFO:teuthology.orchestra.run.smithi163.stderr:+ systemctl stop nfs-server 2024-04-15T22:25:09.764 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:25:09.772 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:25:09.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs volume create foofs' 2024-04-15T22:25:11.017 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:10 smithi081 ceph-mon[24860]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:11.017 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:10 smithi081 ceph-mon[24860]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:11.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:10 smithi163 ceph-mon[27043]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:11.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:10 smithi163 ceph-mon[27043]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:12.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:12 smithi081 ceph-mon[24860]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:12.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:12 smithi081 ceph-mon[24860]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:12.555 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:12 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-15T22:25:12.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:12 smithi163 ceph-mon[27043]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:12.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:12 smithi163 ceph-mon[27043]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:12.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:12 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081[24856]: 2024-04-15T22:25:13.011+0000 7fe9387e4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-15T22:25:13.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: fsmap foofs:0 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi081.qxeoot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi081.qxeoot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:25:13.306 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:13 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:13.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-15T22:25:13.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T22:25:13.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-15T22:25:13.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-15T22:25:13.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T22:25:13.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: fsmap foofs:0 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi081.qxeoot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi081.qxeoot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:25:13.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:13 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:13.713 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-15T22:25:13.723 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-15T22:25:13.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch ls -f json 2024-04-15T22:25:14.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:14 smithi081 ceph-mon[24860]: Saving service mds.foofs spec with placement count:2 2024-04-15T22:25:14.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:14 smithi081 ceph-mon[24860]: Deploying daemon mds.foofs.smithi081.qxeoot on smithi081 2024-04-15T22:25:14.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:14 smithi081 ceph-mon[24860]: pgmap v213: 65 pgs: 9 creating+peering, 7 active+clean, 49 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:14.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:14 smithi081 ceph-mon[24860]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T22:25:14.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:14 smithi163 ceph-mon[27043]: Saving service mds.foofs spec with placement count:2 2024-04-15T22:25:14.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:14 smithi163 ceph-mon[27043]: Deploying daemon mds.foofs.smithi081.qxeoot on smithi081 2024-04-15T22:25:14.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:14 smithi163 ceph-mon[27043]: pgmap v213: 65 pgs: 9 creating+peering, 7 active+clean, 49 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:14.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:14 smithi163 ceph-mon[27043]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T22:25:16.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:16 smithi081 ceph-mon[24860]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T22:25:16.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:16 smithi081 ceph-mon[24860]: pgmap v216: 65 pgs: 9 creating+peering, 27 active+clean, 29 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:16.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:16 smithi163 ceph-mon[27043]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T22:25:16.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:16 smithi163 ceph-mon[27043]: pgmap v216: 65 pgs: 9 creating+peering, 27 active+clean, 29 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:16.928 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:25:16.928 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-15T22:18:43.469303Z", "last_refresh": "2024-04-15T22:22:21.621168Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-15T22:18:37.288554Z", "last_refresh": "2024-04-15T22:22:21.621230Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-15T22:18:40.261405Z", "last_refresh": "2024-04-15T22:22:21.621290Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-15T22:25:13.033331Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-15T22:25:13.026882Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-15T22:18:35.422263Z", "last_refresh": "2024-04-15T22:22:21.621081Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi081:172.21.15.81=smithi081", "smithi163:172.21.15.163=smithi163"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-15T22:19:27.240461Z", "last_refresh": "2024-04-15T22:22:21.620942Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-15T22:18:42.087088Z", "last_refresh": "2024-04-15T22:22:21.621349Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-15T22:22:21.621469Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-15T22:18:38.497224Z", "last_refresh": "2024-04-15T22:22:21.621410Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-15T22:25:17.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:17 smithi081 ceph-mon[24860]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:17.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:17 smithi163 ceph-mon[27043]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:18.279 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-15T22:25:18.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:18.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi163.xhjdei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:25:18.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi163.xhjdei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:25:18.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:18.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: Deploying daemon mds.foofs.smithi163.xhjdei on smithi163 2024-04-15T22:25:18.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:18.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: pgmap v217: 65 pgs: 4 creating+peering, 57 active+clean, 4 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: mds.? [v2:172.21.15.81:6834/3630172207,v1:172.21.15.81:6835/3630172207] up:boot 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: daemon mds.foofs.smithi081.qxeoot assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: Cluster is now healthy 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: fsmap foofs:0 1 up:standby 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi081.qxeoot"}]: dispatch 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:creating} 2024-04-15T22:25:18.678 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:18 smithi163 ceph-mon[27043]: daemon mds.foofs.smithi081.qxeoot is now active in filesystem foofs as rank 0 2024-04-15T22:25:19.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi163.xhjdei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi163.xhjdei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: Deploying daemon mds.foofs.smithi163.xhjdei on smithi163 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: pgmap v217: 65 pgs: 4 creating+peering, 57 active+clean, 4 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: mds.? [v2:172.21.15.81:6834/3630172207,v1:172.21.15.81:6835/3630172207] up:boot 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: daemon mds.foofs.smithi081.qxeoot assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: Cluster is now healthy 2024-04-15T22:25:19.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: fsmap foofs:0 1 up:standby 2024-04-15T22:25:19.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi081.qxeoot"}]: dispatch 2024-04-15T22:25:19.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:creating} 2024-04-15T22:25:19.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:18 smithi081 ceph-mon[24860]: daemon mds.foofs.smithi081.qxeoot is now active in filesystem foofs as rank 0 2024-04-15T22:25:19.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch ls -f json 2024-04-15T22:25:20.172 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:20 smithi081 ceph-mon[24860]: mds.? [v2:172.21.15.81:6834/3630172207,v1:172.21.15.81:6835/3630172207] up:active 2024-04-15T22:25:20.172 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:20 smithi081 ceph-mon[24860]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:active} 2024-04-15T22:25:20.173 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:20 smithi081 ceph-mon[24860]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 306 B/s wr, 0 op/s 2024-04-15T22:25:20.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:20 smithi163 ceph-mon[27043]: mds.? [v2:172.21.15.81:6834/3630172207,v1:172.21.15.81:6835/3630172207] up:active 2024-04-15T22:25:20.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:20 smithi163 ceph-mon[27043]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:active} 2024-04-15T22:25:20.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:20 smithi163 ceph-mon[27043]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 306 B/s wr, 0 op/s 2024-04-15T22:25:21.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: mds.? [v2:172.21.15.163:6832/3723992295,v1:172.21.15.163:6833/3723992295] up:boot 2024-04-15T22:25:21.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:active} 1 up:standby 2024-04-15T22:25:21.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi163.xhjdei"}]: dispatch 2024-04-15T22:25:21.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:active} 1 up:standby 2024-04-15T22:25:21.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:21.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:21.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:21.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:21 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:21.347 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:25:21.347 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-15T22:18:43.469303Z", "last_refresh": "2024-04-15T22:22:21.621168Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-15T22:18:37.288554Z", "last_refresh": "2024-04-15T22:22:21.621230Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-15T22:18:40.261405Z", "last_refresh": "2024-04-15T22:22:21.621290Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-15T22:25:13.033331Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-15T22:25:13.026882Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-15T22:18:35.422263Z", "last_refresh": "2024-04-15T22:22:21.621081Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi081:172.21.15.81=smithi081", "smithi163:172.21.15.163=smithi163"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-15T22:19:27.240461Z", "last_refresh": "2024-04-15T22:22:21.620942Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-15T22:18:42.087088Z", "last_refresh": "2024-04-15T22:22:21.621349Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-15T22:22:21.621469Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-15T22:18:38.497224Z", "last_refresh": "2024-04-15T22:22:21.621410Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-15T22:25:21.423 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: mds.? [v2:172.21.15.163:6832/3723992295,v1:172.21.15.163:6833/3723992295] up:boot 2024-04-15T22:25:21.423 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:active} 1 up:standby 2024-04-15T22:25:21.423 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi163.xhjdei"}]: dispatch 2024-04-15T22:25:21.423 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: fsmap foofs:1 {0=foofs.smithi081.qxeoot=up:active} 1 up:standby 2024-04-15T22:25:21.423 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:21.423 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:21.424 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:21.424 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:21 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:22.123 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-15T22:25:22.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:22 smithi081 ceph-mon[24860]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:22.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:22 smithi081 ceph-mon[24860]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-04-15T22:25:22.387 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:22 smithi163 ceph-mon[27043]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:22.387 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:22 smithi163 ceph-mon[27043]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-04-15T22:25:23.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch ls -f json 2024-04-15T22:25:23.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:23 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:23.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:23 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:23.443 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:23 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:23.443 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:23 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:24.772 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:24 smithi081 ceph-mon[24860]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-15T22:25:24.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:24 smithi163 ceph-mon[27043]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-15T22:25:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:25:25.219 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-15T22:18:43.469303Z", "last_refresh": "2024-04-15T22:22:21.621168Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-15T22:18:37.288554Z", "last_refresh": "2024-04-15T22:22:21.621230Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-15T22:18:40.261405Z", "last_refresh": "2024-04-15T22:22:21.621290Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-15T22:25:13.033331Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-15T22:25:13.026882Z", "last_refresh": "2024-04-15T22:25:22.100184Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-15T22:18:35.422263Z", "last_refresh": "2024-04-15T22:22:21.621081Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi081:172.21.15.81=smithi081", "smithi163:172.21.15.163=smithi163"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-15T22:19:27.240461Z", "last_refresh": "2024-04-15T22:22:21.620942Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-15T22:18:42.087088Z", "last_refresh": "2024-04-15T22:22:21.621349Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-15T22:22:21.621469Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-15T22:18:38.497224Z", "last_refresh": "2024-04-15T22:22:21.621410Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-15T22:25:26.486 INFO:tasks.cephadm:mds.foofs has 1/2 2024-04-15T22:25:27.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:26 smithi081 ceph-mon[24860]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:27.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:26 smithi081 ceph-mon[24860]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-15T22:25:27.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:26 smithi163 ceph-mon[27043]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:27.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:26 smithi163 ceph-mon[27043]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-15T22:25:27.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- ceph orch ls -f json 2024-04-15T22:25:29.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:28 smithi081 ceph-mon[24860]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-15T22:25:29.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:28 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:29.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:28 smithi163 ceph-mon[27043]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-15T22:25:29.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:28 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:25:29.836 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-15T22:18:43.469303Z", "last_refresh": "2024-04-15T22:25:29.694453Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-15T22:18:37.288554Z", "last_refresh": "2024-04-15T22:25:22.099267Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-15T22:18:40.261405Z", "last_refresh": "2024-04-15T22:25:29.694581Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-15T22:25:13.033331Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-15T22:25:13.026882Z", "last_refresh": "2024-04-15T22:25:22.100184Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-15T22:18:35.422263Z", "last_refresh": "2024-04-15T22:25:22.099486Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi081:172.21.15.81=smithi081", "smithi163:172.21.15.163=smithi163"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-15T22:19:27.240461Z", "last_refresh": "2024-04-15T22:25:22.099621Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-15T22:18:42.087088Z", "last_refresh": "2024-04-15T22:25:22.099728Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-15T22:25:22.099802Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-15T22:18:38.497224Z", "last_refresh": "2024-04-15T22:25:29.694706Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-15T22:25:30.599 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-15T22:25:30.599 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:25:30.608 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:25:30.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:30.920 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:30 smithi081 ceph-mon[24860]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:31.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:30 smithi163 ceph-mon[27043]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:25:33.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:32 smithi163 ceph-mon[27043]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 938 B/s wr, 3 op/s 2024-04-15T22:25:33.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:32 smithi081 ceph-mon[24860]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 938 B/s wr, 3 op/s 2024-04-15T22:25:33.907 INFO:teuthology.orchestra.run.smithi081.stdout:NFS Cluster Created Successfully 2024-04-15T22:25:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:33 smithi163 ceph-mon[27043]: from='client.14504 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:33 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-15T22:25:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:33 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:33 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-15T22:25:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:33 smithi163 ceph-mon[27043]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T22:25:34.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:33 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-15T22:25:34.289 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:33 smithi081 ceph-mon[24860]: from='client.14504 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:34.289 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:33 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-15T22:25:34.289 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:33 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:34.289 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:33 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-15T22:25:34.289 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:33 smithi081 ceph-mon[24860]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T22:25:34.289 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:33 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-15T22:25:34.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-15T22:25:35.134 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: pgmap v226: 97 pgs: 5 creating+peering, 27 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:35.134 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-15T22:25:35.134 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: Saving service nfs.foo spec with placement count:2 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: Creating key for client.nfs.foo.0.0.smithi081.aftucs 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-15T22:25:35.135 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:35.136 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-15T22:25:35.136 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-15T22:25:35.136 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:35.136 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:35.136 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:34 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: pgmap v226: 97 pgs: 5 creating+peering, 27 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: Saving service nfs.foo spec with placement count:2 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:35.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: Creating key for client.nfs.foo.0.0.smithi081.aftucs 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi081.aftucs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:35.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:34 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:36.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:35 smithi081 ceph-mon[24860]: Rados config object exists: conf-nfs.foo 2024-04-15T22:25:36.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:35 smithi081 ceph-mon[24860]: Creating key for client.nfs.foo.0.0.smithi081.aftucs-rgw 2024-04-15T22:25:36.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:35 smithi081 ceph-mon[24860]: Deploying daemon nfs.foo.0.0.smithi081.aftucs on smithi081 2024-04-15T22:25:36.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:35 smithi081 ceph-mon[24860]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T22:25:36.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:35 smithi081 ceph-mon[24860]: mgrmap e20: smithi081.wfvsja(active, since 5m), standbys: smithi163.cdbjip 2024-04-15T22:25:36.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:35 smithi163 ceph-mon[27043]: Rados config object exists: conf-nfs.foo 2024-04-15T22:25:36.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:35 smithi163 ceph-mon[27043]: Creating key for client.nfs.foo.0.0.smithi081.aftucs-rgw 2024-04-15T22:25:36.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:35 smithi163 ceph-mon[27043]: Deploying daemon nfs.foo.0.0.smithi081.aftucs on smithi081 2024-04-15T22:25:36.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:35 smithi163 ceph-mon[27043]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T22:25:36.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:35 smithi163 ceph-mon[27043]: mgrmap e20: smithi081.wfvsja(active, since 5m), standbys: smithi163.cdbjip 2024-04-15T22:25:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:36 smithi163 ceph-mon[27043]: pgmap v229: 97 pgs: 5 creating+peering, 22 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:37.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:36 smithi081 ceph-mon[24860]: pgmap v229: 97 pgs: 5 creating+peering, 22 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:38.397 INFO:teuthology.orchestra.run.smithi081.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-15T22:25:38.397 INFO:teuthology.orchestra.run.smithi081.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-15T22:25:38.397 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-04-15T22:25:38.732 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-15T22:25:38.733 INFO:teuthology.orchestra.run.smithi081.stdout: "bind": "/fake", 2024-04-15T22:25:38.733 INFO:teuthology.orchestra.run.smithi081.stdout: "fs": "foofs", 2024-04-15T22:25:38.733 INFO:teuthology.orchestra.run.smithi081.stdout: "path": "/", 2024-04-15T22:25:38.733 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster": "foo", 2024-04-15T22:25:38.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mode": "RW" 2024-04-15T22:25:38.733 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-15T22:25:39.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: pgmap v230: 97 pgs: 5 creating+peering, 9 unknown, 83 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-15T22:25:39.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-15T22:25:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-15T22:25:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-15T22:25:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-15T22:25:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-15T22:25:39.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:38 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: pgmap v230: 97 pgs: 5 creating+peering, 9 unknown, 83 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-15T22:25:39.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:38 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-15T22:25:39.456 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-15T22:25:40.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:39 smithi081 ceph-mon[24860]: Creating key for client.nfs.foo.1.0.smithi163.ixkiam 2024-04-15T22:25:40.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:39 smithi081 ceph-mon[24860]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-15T22:25:40.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:39 smithi081 ceph-mon[24860]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:40.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:39 smithi163 ceph-mon[27043]: Creating key for client.nfs.foo.1.0.smithi163.ixkiam 2024-04-15T22:25:40.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:39 smithi163 ceph-mon[27043]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-15T22:25:40.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:39 smithi163 ceph-mon[27043]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:40.978 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:40 smithi081 ceph-mon[24860]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 299 B/s wr, 0 op/s 2024-04-15T22:25:41.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:40 smithi163 ceph-mon[27043]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 299 B/s wr, 0 op/s 2024-04-15T22:25:41.976 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-15T22:25:41.976 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-15T22:25:41.976 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:41.977 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:41.977 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:41 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:42.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-15T22:25:42.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-15T22:25:42.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:42.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi163.ixkiam-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:42.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:41 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:42 smithi163 ceph-mon[27043]: Rados config object exists: conf-nfs.foo 2024-04-15T22:25:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:42 smithi163 ceph-mon[27043]: Creating key for client.nfs.foo.1.0.smithi163.ixkiam-rgw 2024-04-15T22:25:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:42 smithi163 ceph-mon[27043]: Deploying daemon nfs.foo.1.0.smithi163.ixkiam on smithi163 2024-04-15T22:25:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:42 smithi163 ceph-mon[27043]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:43.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:42 smithi163 ceph-mon[27043]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 383 B/s wr, 0 op/s 2024-04-15T22:25:43.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:42 smithi081 ceph-mon[24860]: Rados config object exists: conf-nfs.foo 2024-04-15T22:25:43.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:42 smithi081 ceph-mon[24860]: Creating key for client.nfs.foo.1.0.smithi163.ixkiam-rgw 2024-04-15T22:25:43.304 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:42 smithi081 ceph-mon[24860]: Deploying daemon nfs.foo.1.0.smithi163.ixkiam on smithi163 2024-04-15T22:25:43.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:42 smithi081 ceph-mon[24860]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:43.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:42 smithi081 ceph-mon[24860]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 383 B/s wr, 0 op/s 2024-04-15T22:25:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:43 smithi163 ceph-mon[27043]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:43 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:43 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:43 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:44.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:43 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:44.274 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:43 smithi081 ceph-mon[24860]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:44.275 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:43 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:44.275 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:43 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:44.275 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:43 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:44.275 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:43 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:45.162 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:44 smithi081 ceph-mon[24860]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-15T22:25:45.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:44 smithi163 ceph-mon[27043]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-15T22:25:46.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:45 smithi163 ceph-mon[27043]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:46.305 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:45 smithi081 ceph-mon[24860]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:47.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:46 smithi081 ceph-mon[24860]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:47.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:46 smithi081 ceph-mon[24860]: pgmap v234: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-15T22:25:47.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:46 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:47.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:46 smithi163 ceph-mon[27043]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:47.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:46 smithi163 ceph-mon[27043]: pgmap v234: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-15T22:25:47.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:46 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:48.237 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:48.237 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-15T22:25:48.237 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-15T22:25:48.237 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-15T22:25:48.237 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-04-15T22:25:48.237 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:48.238 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:48.238 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:49.055 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.056 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:48 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-15T22:25:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-04-15T22:25:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-15T22:25:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-04-15T22:25:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-04-15T22:25:49.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:48 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:25:49.404 INFO:teuthology.orchestra.run.smithi081.stdout:nfs.foo ?:2049 2/2 2s ago 15s count:2 2024-04-15T22:25:50.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:49 smithi081 ceph-mon[24860]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:50.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:49 smithi081 ceph-mon[24860]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T22:25:50.172 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-15T22:25:50.183 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:25:50.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-15T22:25:50.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:49 smithi163 ceph-mon[27043]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:50.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:49 smithi163 ceph-mon[27043]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T22:25:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:+ mkdir /mnt/foo 2024-04-15T22:25:50.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-15T22:25:50.297 INFO:teuthology.orchestra.run.smithi081.stderr:++ hostname 2024-04-15T22:25:50.297 INFO:teuthology.orchestra.run.smithi081.stderr:+ mount -t nfs smithi081:/fake /mnt/foo -o sync 2024-04-15T22:25:50.715 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-15T22:25:50.783 INFO:teuthology.orchestra.run.smithi081.stderr:+ echo test 2024-04-15T22:25:50.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-15T22:25:50.859 INFO:teuthology.orchestra.run.smithi081.stderr:+ sync 2024-04-15T22:25:51.093 INFO:teuthology.run_tasks:Running task parallel... 2024-04-15T22:25:51.104 INFO:teuthology.task.parallel:starting parallel... 2024-04-15T22:25:51.105 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-15T22:25:51.106 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-15T22:25:51.106 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:25:51.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-15T22:25:51.107 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-15T22:25:51.107 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-15T22:25:51.108 INFO:teuthology.task.exec:Executing custom commands... 2024-04-15T22:25:51.108 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:25:51.108 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-15T22:25:51.159 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:50 smithi081 ceph-mon[24860]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:51.159 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:50 smithi081 ceph-mon[24860]: pgmap v238: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-15T22:25:51.167 INFO:teuthology.orchestra.run.smithi081.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-15T22:25:51.167 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-15T22:25:51.167 INFO:teuthology.orchestra.run.smithi081.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-15T22:25:51.168 INFO:teuthology.orchestra.run.smithi081.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-15T22:25:51.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:50 smithi163 ceph-mon[27043]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:25:51.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:50 smithi163 ceph-mon[27043]: pgmap v238: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-15T22:25:51.269 INFO:teuthology.orchestra.run.smithi081.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-15T22:25:51.269 INFO:teuthology.orchestra.run.smithi081.stdout:releasing clients 2024-04-15T22:25:52.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 109 24.90 MB/sec warmup 1 sec latency 31.509 ms 2024-04-15T22:25:53.087 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:52 smithi081 ceph-mon[24860]: pgmap v239: 97 pgs: 97 active+clean; 587 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-15T22:25:53.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:52 smithi163 ceph-mon[27043]: pgmap v239: 97 pgs: 97 active+clean; 587 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-15T22:25:53.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 370 33.00 MB/sec warmup 2 sec latency 33.216 ms 2024-04-15T22:25:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 753 35.59 MB/sec warmup 3 sec latency 12.253 ms 2024-04-15T22:25:54.639 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:54 smithi081 ceph-mon[24860]: pgmap v240: 97 pgs: 97 active+clean; 22 MiB data, 54 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 56 op/s 2024-04-15T22:25:54.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:54 smithi163 ceph-mon[27043]: pgmap v240: 97 pgs: 97 active+clean; 22 MiB data, 54 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 56 op/s 2024-04-15T22:25:55.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-15T22:25:55.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 1759 32.85 MB/sec warmup 4 sec latency 10.098 ms 2024-04-15T22:25:56.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 2891 32.58 MB/sec warmup 5 sec latency 7.309 ms 2024-04-15T22:25:56.898 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:56 smithi081 ceph-mon[24860]: pgmap v241: 97 pgs: 97 active+clean; 61 MiB data, 75 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s wr, 166 op/s 2024-04-15T22:25:57.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:56 smithi163 ceph-mon[27043]: pgmap v241: 97 pgs: 97 active+clean; 61 MiB data, 75 MiB used, 715 GiB / 715 GiB avail; 8.0 MiB/s wr, 166 op/s 2024-04-15T22:25:57.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 3796 33.13 MB/sec warmup 6 sec latency 29.070 ms 2024-04-15T22:25:58.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 4515 29.35 MB/sec warmup 7 sec latency 15.590 ms 2024-04-15T22:25:58.914 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:25:58 smithi081 ceph-mon[24860]: pgmap v242: 97 pgs: 97 active+clean; 105 MiB data, 187 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 12 MiB/s wr, 557 op/s 2024-04-15T22:25:59.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:25:58 smithi163 ceph-mon[27043]: pgmap v242: 97 pgs: 97 active+clean; 105 MiB data, 187 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 12 MiB/s wr, 557 op/s 2024-04-15T22:25:59.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 5568 28.54 MB/sec warmup 8 sec latency 12.697 ms 2024-04-15T22:26:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 6454 28.75 MB/sec warmup 9 sec latency 20.027 ms 2024-04-15T22:26:00.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-15T22:26:00.924 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:00 smithi081 ceph-mon[24860]: pgmap v243: 97 pgs: 97 active+clean; 119 MiB data, 226 MiB used, 715 GiB / 715 GiB avail; 189 B/s rd, 12 MiB/s wr, 560 op/s 2024-04-15T22:26:01.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:00 smithi163 ceph-mon[27043]: pgmap v243: 97 pgs: 97 active+clean; 119 MiB data, 226 MiB used, 715 GiB / 715 GiB avail; 189 B/s rd, 12 MiB/s wr, 560 op/s 2024-04-15T22:26:01.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 7360 29.47 MB/sec warmup 10 sec latency 18.087 ms 2024-04-15T22:26:02.270 INFO:teuthology.orchestra.run.smithi081.stdout: 5 8017 27.32 MB/sec warmup 11 sec latency 24.023 ms 2024-04-15T22:26:02.935 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:02 smithi081 ceph-mon[24860]: pgmap v244: 97 pgs: 97 active+clean; 135 MiB data, 285 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 14 MiB/s wr, 852 op/s 2024-04-15T22:26:03.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:02 smithi163 ceph-mon[27043]: pgmap v244: 97 pgs: 97 active+clean; 135 MiB data, 285 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 14 MiB/s wr, 852 op/s 2024-04-15T22:26:03.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 8531 26.31 MB/sec warmup 12 sec latency 41.233 ms 2024-04-15T22:26:04.270 INFO:teuthology.orchestra.run.smithi081.stdout: 5 9268 25.06 MB/sec warmup 13 sec latency 112.042 ms 2024-04-15T22:26:04.652 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:04 smithi081 ceph-mon[24860]: pgmap v245: 97 pgs: 97 active+clean; 140 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-04-15T22:26:04.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:04 smithi163 ceph-mon[27043]: pgmap v245: 97 pgs: 97 active+clean; 140 MiB data, 314 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-04-15T22:26:05.269 INFO:teuthology.orchestra.run.smithi081.stdout: 5 10065 25.47 MB/sec warmup 14 sec latency 21.911 ms 2024-04-15T22:26:05.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph mgr module enable nfs --force' 2024-04-15T22:26:06.275 INFO:teuthology.orchestra.run.smithi081.stdout: 5 10858 25.82 MB/sec warmup 15 sec latency 18.693 ms 2024-04-15T22:26:06.919 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:06 smithi081 ceph-mon[24860]: pgmap v246: 97 pgs: 97 active+clean; 146 MiB data, 353 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 17 MiB/s wr, 1.24k op/s 2024-04-15T22:26:07.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:06 smithi163 ceph-mon[27043]: pgmap v246: 97 pgs: 97 active+clean; 146 MiB data, 353 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 17 MiB/s wr, 1.24k op/s 2024-04-15T22:26:07.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 11384 24.55 MB/sec warmup 16 sec latency 27.068 ms 2024-04-15T22:26:08.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 11931 23.82 MB/sec warmup 17 sec latency 38.273 ms 2024-04-15T22:26:08.927 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:08 smithi081 ceph-mon[24860]: pgmap v247: 97 pgs: 97 active+clean; 148 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 1.29k op/s 2024-04-15T22:26:09.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:08 smithi163 ceph-mon[27043]: pgmap v247: 97 pgs: 97 active+clean; 148 MiB data, 383 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 1.29k op/s 2024-04-15T22:26:09.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 12703 23.36 MB/sec warmup 18 sec latency 33.045 ms 2024-04-15T22:26:10.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 13490 23.36 MB/sec warmup 19 sec latency 25.694 ms 2024-04-15T22:26:10.810 INFO:teuthology.orchestra.run.smithi081.stderr:module 'nfs' is already enabled 2024-04-15T22:26:10.935 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:10 smithi081 ceph-mon[24860]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 417 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 974 op/s 2024-04-15T22:26:10.935 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:10 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/3886559006' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-15T22:26:10.935 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:10 smithi081 ceph-mon[24860]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-15T22:26:11.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:10 smithi163 ceph-mon[27043]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 417 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 974 op/s 2024-04-15T22:26:11.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:10 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/3886559006' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-15T22:26:11.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:10 smithi163 ceph-mon[27043]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-15T22:26:11.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 14320 24.04 MB/sec warmup 20 sec latency 18.866 ms 2024-04-15T22:26:11.943 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:11 smithi081 ceph-mon[24860]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-15T22:26:11.943 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:11 smithi081 ceph-mon[24860]: mgrmap e21: smithi081.wfvsja(active, since 6m), standbys: smithi163.cdbjip 2024-04-15T22:26:11.997 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-15T22:26:12.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:11 smithi163 ceph-mon[27043]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-15T22:26:12.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:11 smithi163 ceph-mon[27043]: mgrmap e21: smithi081.wfvsja(active, since 6m), standbys: smithi163.cdbjip 2024-04-15T22:26:12.278 INFO:teuthology.orchestra.run.smithi081.stdout: 5 14747 23.22 MB/sec warmup 21 sec latency 22.595 ms 2024-04-15T22:26:12.956 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:12 smithi081 ceph-mon[24860]: pgmap v249: 97 pgs: 97 active+clean; 159 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.13k op/s 2024-04-15T22:26:13.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:12 smithi163 ceph-mon[27043]: pgmap v249: 97 pgs: 97 active+clean; 159 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.13k op/s 2024-04-15T22:26:13.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 15295 22.61 MB/sec warmup 22 sec latency 22.249 ms 2024-04-15T22:26:14.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 16196 22.43 MB/sec warmup 23 sec latency 31.345 ms 2024-04-15T22:26:14.725 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:14 smithi081 ceph-mon[24860]: pgmap v250: 97 pgs: 97 active+clean; 160 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 953 op/s 2024-04-15T22:26:14.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:14 smithi163 ceph-mon[27043]: pgmap v250: 97 pgs: 97 active+clean; 160 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 953 op/s 2024-04-15T22:26:15.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 16822 22.08 MB/sec warmup 24 sec latency 38.793 ms 2024-04-15T22:26:15.431 INFO:teuthology.orchestra.run.smithi081.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:26:16.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 17592 22.54 MB/sec warmup 25 sec latency 29.018 ms 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: pgmap v251: 97 pgs: 97 active+clean; 163 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 891 op/s 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:16.490 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:16 smithi081 ceph-mon[24860]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:26:16.676 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-15T22:26:16.676 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:26:16.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-15T22:26:16.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:16.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:26:16.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:16.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:26:16.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:26:16.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:26:16.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: pgmap v251: 97 pgs: 97 active+clean; 163 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 891 op/s 2024-04-15T22:26:16.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:16.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:16.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:16 smithi163 ceph-mon[27043]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:26:17.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 18269 22.44 MB/sec warmup 26 sec latency 14.559 ms 2024-04-15T22:26:18.253 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:26:18.253 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:18.253 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: pgmap v252: 97 pgs: 97 active+clean; 165 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 9.0 MiB/s wr, 785 op/s 2024-04-15T22:26:18.254 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:26:18.254 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:26:18.254 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:26:18.254 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:18.254 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:18 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:18.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 19024 22.08 MB/sec warmup 27 sec latency 39.430 ms 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: pgmap v252: 97 pgs: 97 active+clean; 165 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 9.0 MiB/s wr, 785 op/s 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:18.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:18 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:19.254 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:19 smithi081 ceph-mon[24860]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:26:19.276 INFO:teuthology.orchestra.run.smithi081.stdout: 5 20070 22.07 MB/sec warmup 28 sec latency 18.319 ms 2024-04-15T22:26:19.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:19 smithi163 ceph-mon[27043]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:26:20.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 21069 22.59 MB/sec warmup 29 sec latency 17.084 ms 2024-04-15T22:26:20.670 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:20 smithi081 ceph-mon[24860]: pgmap v253: 97 pgs: 97 active+clean; 174 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 904 op/s 2024-04-15T22:26:20.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:20 smithi163 ceph-mon[27043]: pgmap v253: 97 pgs: 97 active+clean; 174 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 904 op/s 2024-04-15T22:26:21.038 INFO:teuthology.orchestra.run.smithi081.stdout:true 2024-04-15T22:26:21.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 21826 22.64 MB/sec warmup 30 sec latency 17.902 ms 2024-04-15T22:26:21.586 INFO:teuthology.orchestra.run.smithi081.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-15T22:26:22.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 22522 22.22 MB/sec warmup 31 sec latency 18.724 ms 2024-04-15T22:26:22.689 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:22 smithi081 ceph-mon[24860]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:22.690 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:22 smithi081 ceph-mon[24860]: pgmap v254: 97 pgs: 97 active+clean; 182 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-15T22:26:22.690 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:22 smithi081 ceph-mon[24860]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:22.930 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:22 smithi163 ceph-mon[27043]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:22.930 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:22 smithi163 ceph-mon[27043]: pgmap v254: 97 pgs: 97 active+clean; 182 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-15T22:26:22.930 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:22 smithi163 ceph-mon[27043]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:23.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 23116 22.08 MB/sec warmup 32 sec latency 68.685 ms 2024-04-15T22:26:24.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T22:26:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 23820 21.76 MB/sec warmup 33 sec latency 55.311 ms 2024-04-15T22:26:24.698 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:24 smithi081 ceph-mon[24860]: pgmap v255: 97 pgs: 97 active+clean; 190 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-15T22:26:24.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:24 smithi163 ceph-mon[27043]: pgmap v255: 97 pgs: 97 active+clean; 190 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-15T22:26:25.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 24466 21.97 MB/sec warmup 34 sec latency 30.874 ms 2024-04-15T22:26:26.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 25049 21.89 MB/sec warmup 35 sec latency 43.297 ms 2024-04-15T22:26:26.778 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:26 smithi081 ceph-mon[24860]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1k op/s 2024-04-15T22:26:26.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:26 smithi163 ceph-mon[27043]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1k op/s 2024-04-15T22:26:27.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 25414 21.55 MB/sec warmup 36 sec latency 71.228 ms 2024-04-15T22:26:27.382 INFO:teuthology.orchestra.run.smithi081.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:26:27.382 INFO:teuthology.orchestra.run.smithi081.stdout:alertmanager.smithi081 smithi081 *:9093,9094 running (5m) 38s ago 7m 24.9M - ba2b418f427c 41284a394202 2024-04-15T22:26:27.382 INFO:teuthology.orchestra.run.smithi081.stdout:crash.smithi081 smithi081 running (7m) 38s ago 7m 6400k - 17.2.0 e1d6a67b021e 543e2e5d66e2 2024-04-15T22:26:27.382 INFO:teuthology.orchestra.run.smithi081.stdout:crash.smithi163 smithi163 running (6m) 40s ago 6m 6400k - 17.2.0 e1d6a67b021e e2865e86353b 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:grafana.smithi081 smithi081 *:3000 running (5m) 38s ago 6m 52.1M - 8.3.5 dad864ee21e9 a95ecd98442c 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:mds.foofs.smithi081.qxeoot smithi081 running (70s) 38s ago 69s 13.4M - 17.2.0 e1d6a67b021e bea35303be74 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:mds.foofs.smithi163.xhjdei smithi163 running (67s) 40s ago 67s 12.1M - 17.2.0 e1d6a67b021e f8947dde519d 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.smithi081.wfvsja smithi081 *:9283 running (8m) 38s ago 8m 459M - 17.2.0 e1d6a67b021e f996fd46940d 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.smithi163.cdbjip smithi163 *:8443,9283 running (6m) 40s ago 6m 412M - 17.2.0 e1d6a67b021e 37f9a7722efd 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:mon.smithi081 smithi081 running (8m) 38s ago 8m 46.8M 2048M 17.2.0 e1d6a67b021e 8622084b079c 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:mon.smithi163 smithi163 running (6m) 40s ago 6m 37.5M 2048M 17.2.0 e1d6a67b021e 06c7e5f8d2ba 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:nfs.foo.0.0.smithi081.aftucs smithi081 *:2049 running (49s) 38s ago 49s 60.2M - 4.0 e1d6a67b021e cff7a36e0f8e 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:nfs.foo.1.0.smithi163.ixkiam smithi163 *:2049 running (44s) 40s ago 43s 48.6M - 4.0 e1d6a67b021e 8c0537b97e7f 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:node-exporter.smithi081 smithi081 *:9100 running (6m) 38s ago 6m 18.2M - 1dbe0e931976 6205702b4b03 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:node-exporter.smithi163 smithi163 *:9100 running (6m) 40s ago 6m 17.0M - 1dbe0e931976 7c2719210d6d 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 smithi081 running (5m) 38s ago 5m 46.5M 3216M 17.2.0 e1d6a67b021e b7b6a575e49b 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 smithi081 running (5m) 38s ago 5m 45.3M 3216M 17.2.0 e1d6a67b021e c8f49993a72b 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 smithi081 running (4m) 38s ago 4m 40.1M 3216M 17.2.0 e1d6a67b021e cfba92df08d2 2024-04-15T22:26:27.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 smithi081 running (4m) 38s ago 4m 40.6M 3216M 17.2.0 e1d6a67b021e d878f89983b5 2024-04-15T22:26:27.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 smithi163 running (3m) 40s ago 3m 40.6M 3984M 17.2.0 e1d6a67b021e 4e994b835b7e 2024-04-15T22:26:27.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 smithi163 running (3m) 40s ago 3m 38.7M 3984M 17.2.0 e1d6a67b021e a25f44d5b877 2024-04-15T22:26:27.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 smithi163 running (3m) 40s ago 3m 40.0M 3984M 17.2.0 e1d6a67b021e c695e1b9459d 2024-04-15T22:26:27.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 smithi163 running (2m) 40s ago 2m 41.1M 3984M 17.2.0 e1d6a67b021e 8da0636b871f 2024-04-15T22:26:27.384 INFO:teuthology.orchestra.run.smithi081.stdout:prometheus.smithi081 smithi081 *:9095 running (5m) 38s ago 5m 64.0M - 514e6a882f6e 577580326da4 2024-04-15T22:26:28.277 INFO:teuthology.orchestra.run.smithi081.stdout: 5 25932 21.16 MB/sec warmup 37 sec latency 32.269 ms 2024-04-15T22:26:28.540 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:28 smithi081 ceph-mon[24860]: pgmap v257: 97 pgs: 97 active+clean; 194 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-15T22:26:28.541 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:28 smithi081 ceph-mon[24860]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:28.900 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions' 2024-04-15T22:26:28.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:28 smithi163 ceph-mon[27043]: pgmap v257: 97 pgs: 97 active+clean; 194 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-15T22:26:28.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:28 smithi163 ceph-mon[27043]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:26:29.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 26423 20.86 MB/sec warmup 38 sec latency 31.647 ms 2024-04-15T22:26:30.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 27136 20.78 MB/sec warmup 39 sec latency 31.580 ms 2024-04-15T22:26:30.554 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:30 smithi081 ceph-mon[24860]: pgmap v258: 97 pgs: 97 active+clean; 192 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 934 op/s 2024-04-15T22:26:30.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:30 smithi163 ceph-mon[27043]: pgmap v258: 97 pgs: 97 active+clean; 192 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 934 op/s 2024-04-15T22:26:31.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 27706 20.69 MB/sec warmup 40 sec latency 42.848 ms 2024-04-15T22:26:32.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 28437 20.85 MB/sec warmup 41 sec latency 23.165 ms 2024-04-15T22:26:32.375 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-15T22:26:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": { 2024-04-15T22:26:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:26:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:26:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": { 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": { 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": { 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "overall": { 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-15T22:26:32.376 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-15T22:26:32.562 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:32 smithi081 ceph-mon[24860]: pgmap v259: 97 pgs: 97 active+clean; 198 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 861 op/s 2024-04-15T22:26:32.562 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:32 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/2768440621' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:26:32.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:32 smithi163 ceph-mon[27043]: pgmap v259: 97 pgs: 97 active+clean; 198 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 861 op/s 2024-04-15T22:26:32.927 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:32 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/2768440621' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:26:33.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 29082 20.77 MB/sec warmup 42 sec latency 24.132 ms 2024-04-15T22:26:33.592 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-15T22:26:33.823 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:33 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:34.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:33 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:26:34.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 29824 20.59 MB/sec warmup 43 sec latency 24.386 ms 2024-04-15T22:26:34.831 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:34 smithi081 ceph-mon[24860]: pgmap v260: 97 pgs: 97 active+clean; 198 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 859 op/s 2024-04-15T22:26:35.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:34 smithi163 ceph-mon[27043]: pgmap v260: 97 pgs: 97 active+clean; 198 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 859 op/s 2024-04-15T22:26:35.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 30656 20.57 MB/sec warmup 44 sec latency 40.506 ms 2024-04-15T22:26:36.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 31240 20.50 MB/sec warmup 45 sec latency 26.740 ms 2024-04-15T22:26:36.649 INFO:teuthology.orchestra.run.smithi081.stdout:wait for servicemap items w/ changing names to refresh 2024-04-15T22:26:36.852 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:36 smithi081 ceph-mon[24860]: pgmap v261: 97 pgs: 97 active+clean; 206 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 823 op/s 2024-04-15T22:26:37.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:36 smithi163 ceph-mon[27043]: pgmap v261: 97 pgs: 97 active+clean; 206 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 823 op/s 2024-04-15T22:26:37.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 31813 20.57 MB/sec warmup 46 sec latency 29.835 ms 2024-04-15T22:26:37.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'sleep 60' 2024-04-15T22:26:38.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 32320 20.50 MB/sec warmup 47 sec latency 35.625 ms 2024-04-15T22:26:38.831 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:38 smithi081 ceph-mon[24860]: pgmap v262: 97 pgs: 97 active+clean; 216 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 895 op/s 2024-04-15T22:26:39.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:38 smithi163 ceph-mon[27043]: pgmap v262: 97 pgs: 97 active+clean; 216 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 895 op/s 2024-04-15T22:26:39.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 32779 20.18 MB/sec warmup 48 sec latency 22.604 ms 2024-04-15T22:26:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 33366 20.02 MB/sec warmup 49 sec latency 25.061 ms 2024-04-15T22:26:40.847 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:40 smithi081 ceph-mon[24860]: pgmap v263: 97 pgs: 97 active+clean; 219 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.6 MiB/s wr, 792 op/s 2024-04-15T22:26:41.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:40 smithi163 ceph-mon[27043]: pgmap v263: 97 pgs: 97 active+clean; 219 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.6 MiB/s wr, 792 op/s 2024-04-15T22:26:41.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 33928 19.91 MB/sec warmup 50 sec latency 46.058 ms 2024-04-15T22:26:42.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 34643 19.86 MB/sec warmup 51 sec latency 44.542 ms 2024-04-15T22:26:42.902 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:42 smithi081 ceph-mon[24860]: pgmap v264: 97 pgs: 97 active+clean; 219 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 880 op/s 2024-04-15T22:26:43.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:42 smithi163 ceph-mon[27043]: pgmap v264: 97 pgs: 97 active+clean; 219 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 880 op/s 2024-04-15T22:26:43.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 35278 19.99 MB/sec warmup 52 sec latency 31.365 ms 2024-04-15T22:26:44.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 35830 19.96 MB/sec warmup 53 sec latency 45.419 ms 2024-04-15T22:26:44.919 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:44 smithi081 ceph-mon[24860]: pgmap v265: 97 pgs: 97 active+clean; 224 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 9.3 MiB/s wr, 925 op/s 2024-04-15T22:26:45.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:44 smithi163 ceph-mon[27043]: pgmap v265: 97 pgs: 97 active+clean; 224 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 9.3 MiB/s wr, 925 op/s 2024-04-15T22:26:45.282 INFO:teuthology.orchestra.run.smithi081.stdout: 5 36331 19.73 MB/sec warmup 54 sec latency 40.522 ms 2024-04-15T22:26:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: 5 36931 19.58 MB/sec warmup 55 sec latency 21.793 ms 2024-04-15T22:26:46.930 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:46 smithi081 ceph-mon[24860]: pgmap v266: 97 pgs: 97 active+clean; 228 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 9.3 MiB/s wr, 911 op/s 2024-04-15T22:26:47.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:46 smithi163 ceph-mon[27043]: pgmap v266: 97 pgs: 97 active+clean; 228 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 9.3 MiB/s wr, 911 op/s 2024-04-15T22:26:47.284 INFO:teuthology.orchestra.run.smithi081.stdout: 5 37490 19.49 MB/sec warmup 56 sec latency 72.374 ms 2024-04-15T22:26:47.938 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:47.939 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:48.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:48.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:48.283 INFO:teuthology.orchestra.run.smithi081.stdout: 5 38364 19.57 MB/sec warmup 57 sec latency 24.101 ms 2024-04-15T22:26:48.942 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:48 smithi081 ceph-mon[24860]: pgmap v267: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 868 op/s 2024-04-15T22:26:49.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:48 smithi163 ceph-mon[27043]: pgmap v267: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 868 op/s 2024-04-15T22:26:49.284 INFO:teuthology.orchestra.run.smithi081.stdout: 5 39225 19.80 MB/sec warmup 58 sec latency 13.237 ms 2024-04-15T22:26:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 39808 19.66 MB/sec warmup 59 sec latency 43.230 ms 2024-04-15T22:26:50.951 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:50 smithi081 ceph-mon[24860]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.9 MiB/s wr, 730 op/s 2024-04-15T22:26:51.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:50 smithi163 ceph-mon[27043]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.9 MiB/s wr, 730 op/s 2024-04-15T22:26:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 40418 19.53 MB/sec warmup 60 sec latency 56.748 ms 2024-04-15T22:26:52.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 41181 19.48 MB/sec warmup 61 sec latency 18.077 ms 2024-04-15T22:26:52.993 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:52 smithi081 ceph-mon[24860]: pgmap v269: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 906 op/s 2024-04-15T22:26:53.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:52 smithi163 ceph-mon[27043]: pgmap v269: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 906 op/s 2024-04-15T22:26:53.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 41903 19.53 MB/sec warmup 62 sec latency 68.419 ms 2024-04-15T22:26:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 42788 19.75 MB/sec warmup 63 sec latency 19.456 ms 2024-04-15T22:26:55.006 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:54 smithi081 ceph-mon[24860]: pgmap v270: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.2 MiB/s wr, 850 op/s 2024-04-15T22:26:55.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:54 smithi163 ceph-mon[27043]: pgmap v270: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.2 MiB/s wr, 850 op/s 2024-04-15T22:26:55.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 43661 19.74 MB/sec warmup 64 sec latency 12.427 ms 2024-04-15T22:26:56.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 44332 19.64 MB/sec warmup 65 sec latency 62.666 ms 2024-04-15T22:26:56.812 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:56 smithi081 ceph-mon[24860]: pgmap v271: 97 pgs: 97 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.4 MiB/s wr, 805 op/s 2024-04-15T22:26:57.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:56 smithi163 ceph-mon[27043]: pgmap v271: 97 pgs: 97 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.4 MiB/s wr, 805 op/s 2024-04-15T22:26:57.290 INFO:teuthology.orchestra.run.smithi081.stdout: 5 44973 19.54 MB/sec warmup 66 sec latency 22.242 ms 2024-04-15T22:26:58.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 45597 19.61 MB/sec warmup 67 sec latency 23.527 ms 2024-04-15T22:26:58.823 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:26:58 smithi081 ceph-mon[24860]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 915 op/s 2024-04-15T22:26:59.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:26:58 smithi163 ceph-mon[27043]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 915 op/s 2024-04-15T22:26:59.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 46175 19.67 MB/sec warmup 68 sec latency 48.005 ms 2024-04-15T22:27:00.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 46639 19.52 MB/sec warmup 69 sec latency 80.271 ms 2024-04-15T22:27:00.841 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:00 smithi081 ceph-mon[24860]: pgmap v273: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 844 op/s 2024-04-15T22:27:01.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:00 smithi163 ceph-mon[27043]: pgmap v273: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 844 op/s 2024-04-15T22:27:01.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 47097 19.37 MB/sec warmup 70 sec latency 35.935 ms 2024-04-15T22:27:02.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 47763 19.26 MB/sec warmup 71 sec latency 26.017 ms 2024-04-15T22:27:02.871 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:02 smithi081 ceph-mon[24860]: pgmap v274: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-15T22:27:03.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:02 smithi163 ceph-mon[27043]: pgmap v274: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-15T22:27:03.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 48473 19.24 MB/sec warmup 72 sec latency 36.233 ms 2024-04-15T22:27:04.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 49408 19.44 MB/sec warmup 73 sec latency 18.675 ms 2024-04-15T22:27:04.895 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:04 smithi081 ceph-mon[24860]: pgmap v275: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-15T22:27:05.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:04 smithi163 ceph-mon[27043]: pgmap v275: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-15T22:27:05.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 50260 19.50 MB/sec warmup 74 sec latency 14.890 ms 2024-04-15T22:27:06.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 50970 19.44 MB/sec warmup 75 sec latency 21.798 ms 2024-04-15T22:27:06.915 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:06 smithi081 ceph-mon[24860]: pgmap v276: 97 pgs: 97 active+clean; 278 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.04k op/s 2024-04-15T22:27:07.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:06 smithi163 ceph-mon[27043]: pgmap v276: 97 pgs: 97 active+clean; 278 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.04k op/s 2024-04-15T22:27:07.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 51709 19.36 MB/sec warmup 76 sec latency 26.376 ms 2024-04-15T22:27:08.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 52411 19.40 MB/sec warmup 77 sec latency 32.518 ms 2024-04-15T22:27:08.928 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:08 smithi081 ceph-mon[24860]: pgmap v277: 97 pgs: 97 active+clean; 278 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 949 op/s 2024-04-15T22:27:09.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:08 smithi163 ceph-mon[27043]: pgmap v277: 97 pgs: 97 active+clean; 278 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 949 op/s 2024-04-15T22:27:09.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 53175 19.53 MB/sec warmup 78 sec latency 22.866 ms 2024-04-15T22:27:10.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 53721 19.47 MB/sec warmup 79 sec latency 54.454 ms 2024-04-15T22:27:10.952 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:10 smithi081 ceph-mon[24860]: pgmap v278: 97 pgs: 97 active+clean; 283 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 859 op/s 2024-04-15T22:27:11.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:10 smithi163 ceph-mon[27043]: pgmap v278: 97 pgs: 97 active+clean; 283 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 859 op/s 2024-04-15T22:27:11.294 INFO:teuthology.orchestra.run.smithi081.stdout: 5 54225 19.35 MB/sec warmup 80 sec latency 37.256 ms 2024-04-15T22:27:12.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 54877 19.25 MB/sec warmup 81 sec latency 47.923 ms 2024-04-15T22:27:12.974 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:12 smithi081 ceph-mon[24860]: pgmap v279: 97 pgs: 97 active+clean; 290 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 996 op/s 2024-04-15T22:27:13.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:12 smithi163 ceph-mon[27043]: pgmap v279: 97 pgs: 97 active+clean; 290 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 996 op/s 2024-04-15T22:27:13.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 55549 19.23 MB/sec warmup 82 sec latency 56.591 ms 2024-04-15T22:27:14.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 56229 19.26 MB/sec warmup 83 sec latency 29.543 ms 2024-04-15T22:27:15.006 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:14 smithi081 ceph-mon[24860]: pgmap v280: 97 pgs: 97 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 830 op/s 2024-04-15T22:27:15.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:14 smithi163 ceph-mon[27043]: pgmap v280: 97 pgs: 97 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 830 op/s 2024-04-15T22:27:15.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 56934 19.36 MB/sec warmup 84 sec latency 34.404 ms 2024-04-15T22:27:16.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 57530 19.27 MB/sec warmup 85 sec latency 28.154 ms 2024-04-15T22:27:17.016 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:16 smithi081 ceph-mon[24860]: pgmap v281: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 864 op/s 2024-04-15T22:27:17.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:16 smithi163 ceph-mon[27043]: pgmap v281: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 864 op/s 2024-04-15T22:27:17.292 INFO:teuthology.orchestra.run.smithi081.stdout: 5 58440 19.24 MB/sec warmup 86 sec latency 19.696 ms 2024-04-15T22:27:17.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:17 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:17.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:17 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:17.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:17 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:17.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:17 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:27:17.928 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:17 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:27:18.020 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:17 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:18.021 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:17 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:18.021 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:17 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:18.021 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:17 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:27:18.021 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:17 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' 2024-04-15T22:27:18.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 59288 19.30 MB/sec warmup 87 sec latency 22.857 ms 2024-04-15T22:27:19.028 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:18 smithi081 ceph-mon[24860]: pgmap v282: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 892 op/s 2024-04-15T22:27:19.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:18 smithi163 ceph-mon[27043]: pgmap v282: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 892 op/s 2024-04-15T22:27:19.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 60094 19.42 MB/sec warmup 88 sec latency 23.247 ms 2024-04-15T22:27:20.291 INFO:teuthology.orchestra.run.smithi081.stdout: 5 60835 19.43 MB/sec warmup 89 sec latency 27.727 ms 2024-04-15T22:27:21.054 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:20 smithi081 ceph-mon[24860]: pgmap v283: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 951 op/s 2024-04-15T22:27:21.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:20 smithi163 ceph-mon[27043]: pgmap v283: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 951 op/s 2024-04-15T22:27:21.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 61331 19.32 MB/sec warmup 90 sec latency 49.237 ms 2024-04-15T22:27:22.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 61886 19.23 MB/sec warmup 91 sec latency 51.913 ms 2024-04-15T22:27:22.846 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:22 smithi081 ceph-mon[24860]: pgmap v284: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-04-15T22:27:23.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:22 smithi163 ceph-mon[27043]: pgmap v284: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-04-15T22:27:23.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 62360 19.12 MB/sec warmup 92 sec latency 48.051 ms 2024-04-15T22:27:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 63009 19.14 MB/sec warmup 93 sec latency 33.725 ms 2024-04-15T22:27:24.881 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:24 smithi081 ceph-mon[24860]: pgmap v285: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-04-15T22:27:25.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:24 smithi163 ceph-mon[27043]: pgmap v285: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-04-15T22:27:25.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 63544 19.14 MB/sec warmup 94 sec latency 31.224 ms 2024-04-15T22:27:26.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 64042 19.13 MB/sec warmup 95 sec latency 23.405 ms 2024-04-15T22:27:26.903 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:26 smithi081 ceph-mon[24860]: pgmap v286: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 940 op/s 2024-04-15T22:27:27.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:26 smithi163 ceph-mon[27043]: pgmap v286: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 940 op/s 2024-04-15T22:27:27.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 64532 19.03 MB/sec warmup 96 sec latency 24.576 ms 2024-04-15T22:27:28.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 65044 18.95 MB/sec warmup 97 sec latency 39.992 ms 2024-04-15T22:27:28.925 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:28 smithi081 ceph-mon[24860]: pgmap v287: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 865 op/s 2024-04-15T22:27:29.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:28 smithi163 ceph-mon[27043]: pgmap v287: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 865 op/s 2024-04-15T22:27:29.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 65717 18.89 MB/sec warmup 98 sec latency 53.663 ms 2024-04-15T22:27:30.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 66381 18.92 MB/sec warmup 99 sec latency 29.100 ms 2024-04-15T22:27:30.908 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:30 smithi081 ceph-mon[24860]: pgmap v288: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 818 op/s 2024-04-15T22:27:31.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:30 smithi163 ceph-mon[27043]: pgmap v288: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 818 op/s 2024-04-15T22:27:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 67130 18.98 MB/sec warmup 100 sec latency 21.828 ms 2024-04-15T22:27:32.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 67600 18.94 MB/sec warmup 101 sec latency 26.722 ms 2024-04-15T22:27:32.939 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:32 smithi081 ceph-mon[24860]: pgmap v289: 97 pgs: 97 active+clean; 326 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 751 op/s 2024-04-15T22:27:33.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:32 smithi163 ceph-mon[27043]: pgmap v289: 97 pgs: 97 active+clean; 326 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 751 op/s 2024-04-15T22:27:33.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 68127 18.86 MB/sec warmup 102 sec latency 93.315 ms 2024-04-15T22:27:34.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 68700 18.81 MB/sec warmup 103 sec latency 26.604 ms 2024-04-15T22:27:34.978 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:34 smithi081 ceph-mon[24860]: pgmap v290: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 759 op/s 2024-04-15T22:27:35.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:34 smithi163 ceph-mon[27043]: pgmap v290: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 759 op/s 2024-04-15T22:27:35.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 69325 18.73 MB/sec warmup 104 sec latency 26.050 ms 2024-04-15T22:27:36.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 69984 18.76 MB/sec warmup 105 sec latency 35.835 ms 2024-04-15T22:27:36.997 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:36 smithi081 ceph-mon[24860]: pgmap v291: 97 pgs: 97 active+clean; 340 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.5 MiB/s wr, 858 op/s 2024-04-15T22:27:37.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:36 smithi163 ceph-mon[27043]: pgmap v291: 97 pgs: 97 active+clean; 340 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.5 MiB/s wr, 858 op/s 2024-04-15T22:27:37.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 70601 18.78 MB/sec warmup 106 sec latency 27.792 ms 2024-04-15T22:27:38.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 70957 18.70 MB/sec warmup 107 sec latency 38.248 ms 2024-04-15T22:27:39.038 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:38 smithi081 ceph-mon[24860]: pgmap v292: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 927 op/s 2024-04-15T22:27:39.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:38 smithi163 ceph-mon[27043]: pgmap v292: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 927 op/s 2024-04-15T22:27:39.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 71537 18.65 MB/sec warmup 108 sec latency 24.946 ms 2024-04-15T22:27:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 72139 18.65 MB/sec warmup 109 sec latency 24.882 ms 2024-04-15T22:27:41.047 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:40 smithi081 ceph-mon[24860]: pgmap v293: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 8.3 MiB/s wr, 891 op/s 2024-04-15T22:27:41.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:40 smithi163 ceph-mon[27043]: pgmap v293: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 8.3 MiB/s wr, 891 op/s 2024-04-15T22:27:41.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 72790 18.54 MB/sec warmup 110 sec latency 43.349 ms 2024-04-15T22:27:42.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T22:27:42.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 73429 18.57 MB/sec warmup 111 sec latency 23.126 ms 2024-04-15T22:27:43.059 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:42 smithi081 ceph-mon[24860]: pgmap v294: 97 pgs: 97 active+clean; 343 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 940 op/s 2024-04-15T22:27:43.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:42 smithi163 ceph-mon[27043]: pgmap v294: 97 pgs: 97 active+clean; 343 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 940 op/s 2024-04-15T22:27:43.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 74096 18.60 MB/sec warmup 112 sec latency 33.614 ms 2024-04-15T22:27:44.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 74547 18.55 MB/sec warmup 113 sec latency 26.673 ms 2024-04-15T22:27:45.076 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:44 smithi081 ceph-mon[24860]: pgmap v295: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.8 MiB/s wr, 935 op/s 2024-04-15T22:27:45.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:44 smithi163 ceph-mon[27043]: pgmap v295: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.8 MiB/s wr, 935 op/s 2024-04-15T22:27:45.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 75047 18.50 MB/sec warmup 114 sec latency 26.269 ms 2024-04-15T22:27:46.286 INFO:teuthology.orchestra.run.smithi081.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:27:46.286 INFO:teuthology.orchestra.run.smithi081.stdout:alertmanager.smithi081 smithi081 *:9093,9094 running (7m) 117s ago 8m 24.9M - ba2b418f427c 41284a394202 2024-04-15T22:27:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:crash.smithi081 smithi081 running (8m) 117s ago 8m 6400k - 17.2.0 e1d6a67b021e 543e2e5d66e2 2024-04-15T22:27:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:crash.smithi163 smithi163 running (7m) 119s ago 7m 6400k - 17.2.0 e1d6a67b021e e2865e86353b 2024-04-15T22:27:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:grafana.smithi081 smithi081 *:3000 running (6m) 117s ago 8m 52.1M - 8.3.5 dad864ee21e9 a95ecd98442c 2024-04-15T22:27:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:mds.foofs.smithi081.qxeoot smithi081 running (2m) 117s ago 2m 13.4M - 17.2.0 e1d6a67b021e bea35303be74 2024-04-15T22:27:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:mds.foofs.smithi163.xhjdei smithi163 running (2m) 119s ago 2m 12.1M - 17.2.0 e1d6a67b021e f8947dde519d 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.smithi081.wfvsja smithi081 *:9283 running (9m) 117s ago 9m 459M - 17.2.0 e1d6a67b021e f996fd46940d 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.smithi163.cdbjip smithi163 *:8443,9283 running (7m) 119s ago 7m 412M - 17.2.0 e1d6a67b021e 37f9a7722efd 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:mon.smithi081 smithi081 running (9m) 117s ago 9m 46.8M 2048M 17.2.0 e1d6a67b021e 8622084b079c 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:mon.smithi163 smithi163 running (7m) 119s ago 7m 37.5M 2048M 17.2.0 e1d6a67b021e 06c7e5f8d2ba 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:nfs.foo.0.0.smithi081.aftucs smithi081 *:2049 running (2m) 117s ago 2m 60.2M - 4.0 e1d6a67b021e cff7a36e0f8e 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:nfs.foo.1.0.smithi163.ixkiam smithi163 *:2049 running (2m) 119s ago 2m 48.6M - 4.0 e1d6a67b021e 8c0537b97e7f 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:node-exporter.smithi081 smithi081 *:9100 running (8m) 117s ago 8m 18.2M - 1dbe0e931976 6205702b4b03 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:node-exporter.smithi163 smithi163 *:9100 running (7m) 119s ago 7m 17.0M - 1dbe0e931976 7c2719210d6d 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 smithi081 running (6m) 117s ago 6m 46.5M 3216M 17.2.0 e1d6a67b021e b7b6a575e49b 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 smithi081 running (6m) 117s ago 6m 45.3M 3216M 17.2.0 e1d6a67b021e c8f49993a72b 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 smithi081 running (5m) 117s ago 5m 40.1M 3216M 17.2.0 e1d6a67b021e cfba92df08d2 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 smithi081 running (5m) 117s ago 5m 40.6M 3216M 17.2.0 e1d6a67b021e d878f89983b5 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 smithi163 running (5m) 119s ago 5m 40.6M 3984M 17.2.0 e1d6a67b021e 4e994b835b7e 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 smithi163 running (4m) 119s ago 4m 38.7M 3984M 17.2.0 e1d6a67b021e a25f44d5b877 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 smithi163 running (4m) 119s ago 4m 40.0M 3984M 17.2.0 e1d6a67b021e c695e1b9459d 2024-04-15T22:27:46.289 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 smithi163 running (3m) 119s ago 3m 41.1M 3984M 17.2.0 e1d6a67b021e 8da0636b871f 2024-04-15T22:27:46.290 INFO:teuthology.orchestra.run.smithi081.stdout:prometheus.smithi081 smithi081 *:9095 running (7m) 117s ago 7m 64.0M - 514e6a882f6e 577580326da4 2024-04-15T22:27:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 75523 18.47 MB/sec warmup 115 sec latency 41.480 ms 2024-04-15T22:27:47.088 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:46 smithi081 ceph-mon[24860]: pgmap v296: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.7 MiB/s wr, 955 op/s 2024-04-15T22:27:47.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:46 smithi163 ceph-mon[27043]: pgmap v296: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.7 MiB/s wr, 955 op/s 2024-04-15T22:27:47.298 INFO:teuthology.orchestra.run.smithi081.stdout: 5 76242 18.39 MB/sec warmup 116 sec latency 21.367 ms 2024-04-15T22:27:47.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade status' 2024-04-15T22:27:48.094 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:47 smithi081 ceph-mon[24860]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:48.095 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:48.095 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:47 smithi081 ceph-mon[24860]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:48.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:47 smithi163 ceph-mon[27043]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:48.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:47 smithi163 ceph-mon[27043]: from='mgr.14204 172.21.15.81:0/235901542' entity='mgr.smithi081.wfvsja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi081.wfvsja/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:48.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 76734 18.36 MB/sec warmup 117 sec latency 21.381 ms 2024-04-15T22:27:49.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:48 smithi081 ceph-mon[24860]: pgmap v297: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.1 MiB/s wr, 784 op/s 2024-04-15T22:27:49.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:48 smithi163 ceph-mon[27043]: pgmap v297: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.1 MiB/s wr, 784 op/s 2024-04-15T22:27:49.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 77387 18.39 MB/sec warmup 118 sec latency 62.335 ms 2024-04-15T22:27:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 77860 18.37 MB/sec warmup 119 sec latency 74.808 ms 2024-04-15T22:27:51.036 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:50 smithi081 ceph-mon[24860]: pgmap v298: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.8 MiB/s wr, 755 op/s 2024-04-15T22:27:51.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:50 smithi163 ceph-mon[27043]: pgmap v298: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.8 MiB/s wr, 755 op/s 2024-04-15T22:27:51.648 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-15T22:27:51.648 INFO:teuthology.orchestra.run.smithi081.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", 2024-04-15T22:27:51.648 INFO:teuthology.orchestra.run.smithi081.stdout: "in_progress": true, 2024-04-15T22:27:51.648 INFO:teuthology.orchestra.run.smithi081.stdout: "services_complete": [], 2024-04-15T22:27:51.648 INFO:teuthology.orchestra.run.smithi081.stdout: "progress": "", 2024-04-15T22:27:51.648 INFO:teuthology.orchestra.run.smithi081.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-15T22:27:51.648 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-15T22:27:52.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 78750 13.31 MB/sec execute 1 sec latency 38.823 ms 2024-04-15T22:27:53.051 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:52 smithi081 ceph-mon[24860]: pgmap v299: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 845 op/s 2024-04-15T22:27:53.051 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:52 smithi081 ceph-mon[24860]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:53.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:52 smithi163 ceph-mon[27043]: pgmap v299: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 845 op/s 2024-04-15T22:27:53.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:52 smithi163 ceph-mon[27043]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:53.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph health detail' 2024-04-15T22:27:53.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 79173 12.00 MB/sec execute 2 sec latency 41.136 ms 2024-04-15T22:27:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 79768 10.73 MB/sec execute 3 sec latency 29.971 ms 2024-04-15T22:27:55.084 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:54 smithi081 ceph-mon[24860]: pgmap v300: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 698 op/s 2024-04-15T22:27:55.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:54 smithi163 ceph-mon[27043]: pgmap v300: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 698 op/s 2024-04-15T22:27:55.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 80259 12.32 MB/sec execute 4 sec latency 56.167 ms 2024-04-15T22:27:56.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 80926 13.84 MB/sec execute 5 sec latency 35.439 ms 2024-04-15T22:27:57.106 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:56 smithi081 ceph-mon[24860]: pgmap v301: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-15T22:27:57.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:56 smithi163 ceph-mon[27043]: pgmap v301: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-15T22:27:57.298 INFO:teuthology.orchestra.run.smithi081.stdout:HEALTH_WARN Upgrade: failed to pull target image 2024-04-15T22:27:57.299 INFO:teuthology.orchestra.run.smithi081.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:27:57.299 INFO:teuthology.orchestra.run.smithi081.stdout: host smithi081 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T22:27:57.299 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:57.299 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T22:27:57.299 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765: reading manifest a9a752df26c63acad72e1b3569fd79a515ca0765 in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-15T22:27:57.299 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:57.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 81431 14.49 MB/sec execute 6 sec latency 31.825 ms 2024-04-15T22:27:58.123 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:57 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/554756111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T22:27:58.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:57 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/554756111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T22:27:58.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 81861 14.08 MB/sec execute 7 sec latency 49.727 ms 2024-04-15T22:27:58.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions' 2024-04-15T22:27:59.143 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:27:58 smithi081 ceph-mon[24860]: pgmap v302: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 7.4 MiB/s wr, 637 op/s 2024-04-15T22:27:59.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:27:58 smithi163 ceph-mon[27043]: pgmap v302: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 7.4 MiB/s wr, 637 op/s 2024-04-15T22:27:59.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 82244 13.24 MB/sec execute 8 sec latency 34.378 ms 2024-04-15T22:28:00.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 82691 13.27 MB/sec execute 9 sec latency 48.894 ms 2024-04-15T22:28:00.929 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:00 smithi081 ceph-mon[24860]: pgmap v303: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.6 MiB/s wr, 784 op/s 2024-04-15T22:28:01.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:00 smithi163 ceph-mon[27043]: pgmap v303: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.6 MiB/s wr, 784 op/s 2024-04-15T22:28:01.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 83214 12.62 MB/sec execute 10 sec latency 33.233 ms 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": { 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": { 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:28:01.821 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": { 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": { 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: "overall": { 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-15T22:28:01.822 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-15T22:28:01.938 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:01 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/390814060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:02.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:01 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/390814060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:02.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 83695 12.67 MB/sec execute 11 sec latency 27.789 ms 2024-04-15T22:28:02.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-15T22:28:02.948 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:02 smithi081 ceph-mon[24860]: pgmap v304: 97 pgs: 97 active+clean; 328 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 786 op/s 2024-04-15T22:28:03.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:02 smithi163 ceph-mon[27043]: pgmap v304: 97 pgs: 97 active+clean; 328 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 786 op/s 2024-04-15T22:28:03.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 84253 13.15 MB/sec execute 12 sec latency 27.187 ms 2024-04-15T22:28:04.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 84766 13.50 MB/sec execute 13 sec latency 29.922 ms 2024-04-15T22:28:05.000 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:04 smithi081 ceph-mon[24860]: pgmap v305: 97 pgs: 97 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.4 MiB/s wr, 705 op/s 2024-04-15T22:28:05.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 85182 13.29 MB/sec execute 14 sec latency 51.666 ms 2024-04-15T22:28:05.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:04 smithi163 ceph-mon[27043]: pgmap v305: 97 pgs: 97 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.4 MiB/s wr, 705 op/s 2024-04-15T22:28:06.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 85707 13.30 MB/sec execute 15 sec latency 56.034 ms 2024-04-15T22:28:06.639 INFO:teuthology.orchestra.run.smithi081.stdout:true 2024-04-15T22:28:07.014 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:06 smithi081 ceph-mon[24860]: pgmap v306: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.2 MiB/s wr, 722 op/s 2024-04-15T22:28:07.014 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:06 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/1338953140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:07.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 86232 13.44 MB/sec execute 16 sec latency 34.690 ms 2024-04-15T22:28:07.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:06 smithi163 ceph-mon[27043]: pgmap v306: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.2 MiB/s wr, 722 op/s 2024-04-15T22:28:07.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:06 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/1338953140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:07.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-15T22:28:08.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 86762 13.12 MB/sec execute 17 sec latency 43.922 ms 2024-04-15T22:28:09.035 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:08 smithi081 ceph-mon[24860]: pgmap v307: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.3 MiB/s wr, 682 op/s 2024-04-15T22:28:09.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 87181 12.87 MB/sec execute 18 sec latency 35.694 ms 2024-04-15T22:28:09.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:08 smithi163 ceph-mon[27043]: pgmap v307: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.3 MiB/s wr, 682 op/s 2024-04-15T22:28:10.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 87877 13.49 MB/sec execute 19 sec latency 32.544 ms 2024-04-15T22:28:11.062 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:10 smithi081 ceph-mon[24860]: pgmap v308: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.3 MiB/s wr, 744 op/s 2024-04-15T22:28:11.062 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:10 smithi081 ceph-mon[24860]: from='client.? 172.21.15.81:0/35471688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:11.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 88457 13.77 MB/sec execute 20 sec latency 49.629 ms 2024-04-15T22:28:11.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:10 smithi163 ceph-mon[27043]: pgmap v308: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.3 MiB/s wr, 744 op/s 2024-04-15T22:28:11.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:10 smithi163 ceph-mon[27043]: from='client.? 172.21.15.81:0/35471688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:28:11.920 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-15T22:28:12.138 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ec3c4ddbc6724492904164c0f13ec73b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-15T22:28:12.141 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-15T22:28:12.152 INFO:tasks.cephadm:Teardown begin 2024-04-15T22:28:12.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:28:12.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:28:12.247 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-15T22:28:12.248 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T22:28:12.267 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T22:28:12.299 INFO:teuthology.orchestra.run.smithi081.stdout: 5 88841 13.48 MB/sec execute 21 sec latency 30.170 ms 2024-04-15T22:28:12.303 INFO:tasks.cephadm:Stopping all daemons... 2024-04-15T22:28:12.303 INFO:tasks.cephadm.mon.smithi081:Stopping mon.smithi081... 2024-04-15T22:28:12.303 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081 2024-04-15T22:28:12.475 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:12 smithi081 systemd[1]: Stopping Ceph mon.smithi081 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:12.475 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:12 smithi081 bash[57739]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon.smithi081" found: no such container 2024-04-15T22:28:12.726 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:12 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081[24856]: 2024-04-15T22:28:12.589+0000 7fe93dfef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi081 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:12.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:12 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081[24856]: 2024-04-15T22:28:12.589+0000 7fe93dfef700 -1 mon.smithi081@0(leader) e2 *** Got Signal Terminated *** 2024-04-15T22:28:12.727 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:12 smithi081 podman[57750]: 2024-04-15 22:28:12.711915233 +0000 UTC m=+0.289794393 container died 8622084b079c62c9d2612202ad210ceb1f7f951fa56bc884d6e2514f4c0387ae (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, vcs-type=git, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.openshift.tags=base centos centos-stream) 2024-04-15T22:28:13.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 89328 13.53 MB/sec execute 22 sec latency 27.120 ms 2024-04-15T22:28:13.992 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:13 smithi081 podman[57750]: 2024-04-15 22:28:13.841105372 +0000 UTC m=+1.418984532 container cleanup 8622084b079c62c9d2612202ad210ceb1f7f951fa56bc884d6e2514f4c0387ae (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, ceph=True, GIT_CLEAN=True, distribution-scope=public, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T22:28:13.992 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:13 smithi081 bash[57750]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081 2024-04-15T22:28:13.993 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:13 smithi081 bash[57773]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon.smithi081" found: no such container 2024-04-15T22:28:14.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service' 2024-04-15T22:28:14.253 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:14 smithi081 podman[57762]: 2024-04-15 22:28:14.206143712 +0000 UTC m=+1.489079100 container remove 8622084b079c62c9d2612202ad210ceb1f7f951fa56bc884d6e2514f4c0387ae (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi081, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.component=centos-stream-container, version=8, release=754, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream) 2024-04-15T22:28:14.253 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:14 smithi081 systemd[1]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service: Deactivated successfully. 2024-04-15T22:28:14.253 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:14 smithi081 systemd[1]: Stopped Ceph mon.smithi081 for e7d16278-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:28:14.253 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:Apr 15 22:28:14 smithi081 systemd[1]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi081.service: Consumed 9.186s CPU time. 2024-04-15T22:28:14.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 89788 13.51 MB/sec execute 23 sec latency 60.735 ms 2024-04-15T22:28:14.314 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:14.315 INFO:tasks.cephadm.mon.smithi081:Stopped mon.smithi081 2024-04-15T22:28:14.315 INFO:tasks.cephadm.mon.smithi163:Stopping mon.smithi163... 2024-04-15T22:28:14.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi163 2024-04-15T22:28:14.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:14 smithi163 systemd[1]: Stopping Ceph mon.smithi163 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:14.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:14 smithi163 bash[43192]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon.smithi163" found: no such container 2024-04-15T22:28:14.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:14 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi163[27037]: 2024-04-15T22:28:14.519+0000 7f08affe8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi163 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:14.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:14 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi163[27037]: 2024-04-15T22:28:14.519+0000 7f08affe8700 -1 mon.smithi163@1(peon) e2 *** Got Signal Terminated *** 2024-04-15T22:28:14.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:14 smithi163 podman[43203]: 2024-04-15 22:28:14.550781263 +0000 UTC m=+0.155273944 container died 06c7e5f8d2ba304b5d9bdea4bb0b575ff0bb749674d3461bc8c3d4e2f0666d2e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi163, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.buildah.version=1.19.8, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:28:15.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 13.09 MB/sec execute 24 sec latency 772.001 ms 2024-04-15T22:28:15.387 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:15 smithi163 podman[43203]: 2024-04-15 22:28:15.120854227 +0000 UTC m=+0.725346897 container cleanup 06c7e5f8d2ba304b5d9bdea4bb0b575ff0bb749674d3461bc8c3d4e2f0666d2e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi163, ceph=True, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream) 2024-04-15T22:28:15.388 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:15 smithi163 bash[43203]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon-smithi163 2024-04-15T22:28:15.388 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 15 22:28:15 smithi163 bash[43229]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-mon.smithi163" found: no such container 2024-04-15T22:28:15.392 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@mon.smithi163.service' 2024-04-15T22:28:15.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:15.440 INFO:tasks.cephadm.mon.smithi163:Stopped mon.smithi163 2024-04-15T22:28:15.441 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-15T22:28:15.441 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.0 2024-04-15T22:28:15.762 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:15 smithi081 systemd[1]: Stopping Ceph osd.0 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:15.762 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:15 smithi081 bash[57826]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T22:28:16.054 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:15 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0[35313]: 2024-04-15T22:28:15.761+0000 7f3b9aebd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:16.054 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:15 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0[35313]: 2024-04-15T22:28:15.761+0000 7f3b9aebd700 -1 osd.0 69 *** Got signal Terminated *** 2024-04-15T22:28:16.054 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:15 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0[35313]: 2024-04-15T22:28:15.761+0000 7f3b9aebd700 -1 osd.0 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:28:16.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 12.57 MB/sec execute 25 sec latency 1772.066 ms 2024-04-15T22:28:17.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 12.08 MB/sec execute 26 sec latency 2772.096 ms 2024-04-15T22:28:18.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 11.64 MB/sec execute 27 sec latency 3772.123 ms 2024-04-15T22:28:19.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 11.22 MB/sec execute 28 sec latency 4772.177 ms 2024-04-15T22:28:20.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 10.83 MB/sec execute 29 sec latency 5772.207 ms 2024-04-15T22:28:21.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 10.47 MB/sec execute 30 sec latency 6772.496 ms 2024-04-15T22:28:21.589 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:21 smithi081 podman[57840]: 2024-04-15 22:28:21.30836072 +0000 UTC m=+5.764309077 container died b7b6a575e49ba443401783b731c66d086ed9697393d7db6ec2b641349bbaa9a3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, name=centos-stream, ceph=True, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-04-15T22:28:21.957 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:21 smithi081 podman[57840]: 2024-04-15 22:28:21.690172589 +0000 UTC m=+6.146120956 container cleanup b7b6a575e49ba443401783b731c66d086ed9697393d7db6ec2b641349bbaa9a3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, version=8, name=centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, architecture=x86_64, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=) 2024-04-15T22:28:21.957 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:21 smithi081 bash[57840]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0 2024-04-15T22:28:21.957 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:21 smithi081 bash[57869]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T22:28:22.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 10.13 MB/sec execute 31 sec latency 7772.555 ms 2024-04-15T22:28:22.304 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:21 smithi081 podman[57858]: 2024-04-15 22:28:21.958100479 +0000 UTC m=+0.634830827 container remove b7b6a575e49ba443401783b731c66d086ed9697393d7db6ec2b641349bbaa9a3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, release=754, name=centos-stream, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, ceph=True, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T22:28:22.304 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:22 smithi081 podman[57933]: 2024-04-15 22:28:22.043797775 +0000 UTC m=+0.020580198 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:28:22.304 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:22 smithi081 podman[57933]: 2024-04-15 22:28:22.175962097 +0000 UTC m=+0.152744509 container create 88928cdd4f77573091cf343efe917d45d982d26c7c334f181fac0868f76c888a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, ceph=True, architecture=x86_64, release=754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, RELEASE=HEAD) 2024-04-15T22:28:22.775 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:22 smithi081 podman[57933]: 2024-04-15 22:28:22.428476106 +0000 UTC m=+0.405258518 container init 88928cdd4f77573091cf343efe917d45d982d26c7c334f181fac0868f76c888a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, release=754, ceph=True, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-type=git) 2024-04-15T22:28:22.776 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:22 smithi081 podman[57933]: 2024-04-15 22:28:22.434103066 +0000 UTC m=+0.410885483 container start 88928cdd4f77573091cf343efe917d45d982d26c7c334f181fac0868f76c888a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, version=8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD) 2024-04-15T22:28:22.776 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:22 smithi081 podman[57933]: 2024-04-15 22:28:22.528458032 +0000 UTC m=+0.505240444 container attach 88928cdd4f77573091cf343efe917d45d982d26c7c334f181fac0868f76c888a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, ceph=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_CLEAN=True, version=8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:28:23.054 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 15 22:28:22 smithi081 podman[57933]: 2024-04-15 22:28:22.776998486 +0000 UTC m=+0.753780896 container died 88928cdd4f77573091cf343efe917d45d982d26c7c334f181fac0868f76c888a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, RELEASE=HEAD, version=8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T22:28:23.300 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 9.82 MB/sec execute 32 sec latency 8772.607 ms 2024-04-15T22:28:23.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-15T22:28:23.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:23.528 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-15T22:28:23.528 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-15T22:28:23.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.1 2024-04-15T22:28:23.691 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:23 smithi081 systemd[1]: Stopping Ceph osd.1 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:24.054 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:23 smithi081 bash[58013]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T22:28:24.054 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:23 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1[39392]: 2024-04-15T22:28:23.802+0000 7f063c826700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:24.054 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:23 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1[39392]: 2024-04-15T22:28:23.802+0000 7f063c826700 -1 osd.1 69 *** Got signal Terminated *** 2024-04-15T22:28:24.054 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:23 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1[39392]: 2024-04-15T22:28:23.802+0000 7f063c826700 -1 osd.1 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:28:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 9.52 MB/sec execute 33 sec latency 9772.653 ms 2024-04-15T22:28:25.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 9.24 MB/sec execute 34 sec latency 10772.688 ms 2024-04-15T22:28:26.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 8.98 MB/sec execute 35 sec latency 11772.713 ms 2024-04-15T22:28:27.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 8.73 MB/sec execute 36 sec latency 12772.763 ms 2024-04-15T22:28:28.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 8.49 MB/sec execute 37 sec latency 13772.814 ms 2024-04-15T22:28:29.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 8.27 MB/sec execute 38 sec latency 14772.848 ms 2024-04-15T22:28:29.576 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:29 smithi081 podman[58024]: 2024-04-15 22:28:29.317395318 +0000 UTC m=+5.614938941 container died c8f49993a72bc19dfb5ba300800b04e5115c970961390fef3d0b4f1f73c983f7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, release=754, io.buildah.version=1.19.8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, distribution-scope=public, RELEASE=HEAD, vcs-type=git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T22:28:29.976 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:29 smithi081 podman[58024]: 2024-04-15 22:28:29.676631026 +0000 UTC m=+5.974174645 container cleanup c8f49993a72bc19dfb5ba300800b04e5115c970961390fef3d0b4f1f73c983f7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1, io.openshift.expose-services=, release=754, RELEASE=HEAD, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T22:28:29.976 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:29 smithi081 bash[58024]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1 2024-04-15T22:28:29.977 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:29 smithi081 bash[58057]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T22:28:30.237 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:29 smithi081 podman[58043]: 2024-04-15 22:28:29.97791396 +0000 UTC m=+0.662586271 container remove c8f49993a72bc19dfb5ba300800b04e5115c970961390fef3d0b4f1f73c983f7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vcs-type=git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., RELEASE=HEAD) 2024-04-15T22:28:30.237 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:30 smithi081 podman[58122]: 2024-04-15 22:28:30.099873197 +0000 UTC m=+0.019134487 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:28:30.237 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:30 smithi081 podman[58122]: 2024-04-15 22:28:30.238644159 +0000 UTC m=+0.157905427 container create 98f7758fb03d30dd0f66f7613d1feda2810e3e86eda1ead06d9a86f669a23ec2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, name=centos-stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, com.redhat.component=centos-stream-container, ceph=True) 2024-04-15T22:28:30.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 8.06 MB/sec execute 39 sec latency 15772.916 ms 2024-04-15T22:28:30.554 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:30 smithi081 podman[58122]: 2024-04-15 22:28:30.465927917 +0000 UTC m=+0.385189186 container init 98f7758fb03d30dd0f66f7613d1feda2810e3e86eda1ead06d9a86f669a23ec2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, distribution-scope=public, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, RELEASE=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-15T22:28:30.554 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:30 smithi081 podman[58122]: 2024-04-15 22:28:30.469708972 +0000 UTC m=+0.388970245 container start 98f7758fb03d30dd0f66f7613d1feda2810e3e86eda1ead06d9a86f669a23ec2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-type=git, maintainer=Guillaume Abrioux , release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, version=8) 2024-04-15T22:28:31.054 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:30 smithi081 podman[58122]: 2024-04-15 22:28:30.566889667 +0000 UTC m=+0.486150950 container attach 98f7758fb03d30dd0f66f7613d1feda2810e3e86eda1ead06d9a86f669a23ec2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, maintainer=Guillaume Abrioux , vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, ceph=True, GIT_CLEAN=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:28:31.054 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 15 22:28:30 smithi081 podman[58122]: 2024-04-15 22:28:30.812675216 +0000 UTC m=+0.731936490 container died 98f7758fb03d30dd0f66f7613d1feda2810e3e86eda1ead06d9a86f669a23ec2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, release=754, version=8, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, RELEASE=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git) 2024-04-15T22:28:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 7.85 MB/sec execute 40 sec latency 16772.960 ms 2024-04-15T22:28:31.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-15T22:28:31.705 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:31.705 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-15T22:28:31.705 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-15T22:28:31.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.2 2024-04-15T22:28:32.033 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:31 smithi081 systemd[1]: Stopping Ceph osd.2 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:32.034 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:31 smithi081 bash[58200]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T22:28:32.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 7.66 MB/sec execute 41 sec latency 17772.988 ms 2024-04-15T22:28:32.303 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:32 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2[43080]: 2024-04-15T22:28:32.034+0000 7fe63228b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:32.304 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:32 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2[43080]: 2024-04-15T22:28:32.034+0000 7fe63228b700 -1 osd.2 69 *** Got signal Terminated *** 2024-04-15T22:28:32.304 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:32 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2[43080]: 2024-04-15T22:28:32.034+0000 7fe63228b700 -1 osd.2 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:28:33.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 7.48 MB/sec execute 42 sec latency 18773.038 ms 2024-04-15T22:28:34.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 7.31 MB/sec execute 43 sec latency 19773.103 ms 2024-04-15T22:28:35.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 7.14 MB/sec execute 44 sec latency 20773.166 ms 2024-04-15T22:28:36.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.98 MB/sec execute 45 sec latency 21773.217 ms 2024-04-15T22:28:37.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.83 MB/sec execute 46 sec latency 22773.241 ms 2024-04-15T22:28:37.772 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:37 smithi081 podman[58213]: 2024-04-15 22:28:37.496025558 +0000 UTC m=+5.632759792 container died cfba92df08d208aa2d77182074fb3943b240fd713a2c7ec369645a0d7d7f7839 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.expose-services=, name=centos-stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, RELEASE=HEAD) 2024-04-15T22:28:38.054 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:37 smithi081 podman[58213]: 2024-04-15 22:28:37.848072886 +0000 UTC m=+5.984807110 container cleanup cfba92df08d208aa2d77182074fb3943b240fd713a2c7ec369645a0d7d7f7839 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2, io.openshift.expose-services=, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , version=8, GIT_BRANCH=HEAD) 2024-04-15T22:28:38.054 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:37 smithi081 bash[58213]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2 2024-04-15T22:28:38.054 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:37 smithi081 bash[58247]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T22:28:38.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.68 MB/sec execute 47 sec latency 23773.311 ms 2024-04-15T22:28:38.457 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:38 smithi081 podman[58234]: 2024-04-15 22:28:38.132811809 +0000 UTC m=+0.632768288 container remove cfba92df08d208aa2d77182074fb3943b240fd713a2c7ec369645a0d7d7f7839 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_CLEAN=True, distribution-scope=public, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vendor=Red Hat, Inc.) 2024-04-15T22:28:38.458 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:38 smithi081 podman[58313]: 2024-04-15 22:28:38.210778385 +0000 UTC m=+0.020600470 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:28:38.458 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:38 smithi081 podman[58313]: 2024-04-15 22:28:38.325405571 +0000 UTC m=+0.135227654 container create d3033c2b731a6656836b2b8bb92bbf6f50fecc5e9e185edd4d06f6cf2111174c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, name=centos-stream, architecture=x86_64, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.openshift.expose-services=, release=754, vcs-type=git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:28:38.804 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:38 smithi081 podman[58313]: 2024-04-15 22:28:38.56149449 +0000 UTC m=+0.371316580 container init d3033c2b731a6656836b2b8bb92bbf6f50fecc5e9e185edd4d06f6cf2111174c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., architecture=x86_64, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, name=centos-stream, ceph=True, version=8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux ) 2024-04-15T22:28:38.805 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:38 smithi081 podman[58313]: 2024-04-15 22:28:38.568294879 +0000 UTC m=+0.378116977 container start d3033c2b731a6656836b2b8bb92bbf6f50fecc5e9e185edd4d06f6cf2111174c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , release=754, version=8, vendor=Red Hat, Inc., RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-04-15T22:28:38.805 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:38 smithi081 podman[58313]: 2024-04-15 22:28:38.652897337 +0000 UTC m=+0.462719440 container attach d3033c2b731a6656836b2b8bb92bbf6f50fecc5e9e185edd4d06f6cf2111174c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, io.openshift.expose-services=, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-type=git, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, distribution-scope=public, name=centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T22:28:39.195 INFO:journalctl@ceph.osd.2.smithi081.stdout:Apr 15 22:28:38 smithi081 podman[58313]: 2024-04-15 22:28:38.903382652 +0000 UTC m=+0.713204733 container died d3033c2b731a6656836b2b8bb92bbf6f50fecc5e9e185edd4d06f6cf2111174c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-type=git, ceph=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, version=8, io.k8s.display-name=CentOS Stream 8) 2024-04-15T22:28:39.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.55 MB/sec execute 48 sec latency 24773.337 ms 2024-04-15T22:28:39.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-15T22:28:39.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:39.598 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-15T22:28:39.599 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-15T22:28:39.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.3 2024-04-15T22:28:39.766 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:39 smithi081 systemd[1]: Stopping Ceph osd.3 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:40.054 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:39 smithi081 bash[58393]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T22:28:40.055 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:39 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3[46351]: 2024-04-15T22:28:39.880+0000 7fbf98d0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:40.055 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:39 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3[46351]: 2024-04-15T22:28:39.880+0000 7fbf98d0f700 -1 osd.3 69 *** Got signal Terminated *** 2024-04-15T22:28:40.055 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:39 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3[46351]: 2024-04-15T22:28:39.880+0000 7fbf98d0f700 -1 osd.3 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:28:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.41 MB/sec execute 49 sec latency 25773.389 ms 2024-04-15T22:28:41.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.28 MB/sec execute 50 sec latency 26773.436 ms 2024-04-15T22:28:42.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.16 MB/sec execute 51 sec latency 27773.498 ms 2024-04-15T22:28:42.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:42 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:42.299+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:43.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 6.04 MB/sec execute 52 sec latency 28773.558 ms 2024-04-15T22:28:43.426 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:43 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:43.311+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:43.427 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:43 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:43.074+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:43.513 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:43 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3[46351]: 2024-04-15T22:28:43.241+0000 7fbf926a4700 -1 osd.3 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.467487+0000 front 2024-04-15T22:28:19.467508+0000 (oldest deadline 2024-04-15T22:28:42.367556+0000) 2024-04-15T22:28:44.176 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:44 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:44.118+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:44.177 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:43 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:43.909+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:44.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.93 MB/sec execute 53 sec latency 29773.607 ms 2024-04-15T22:28:44.554 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:44 smithi081 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3[46351]: 2024-04-15T22:28:44.254+0000 7fbf926a4700 -1 osd.3 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.467487+0000 front 2024-04-15T22:28:19.467508+0000 (oldest deadline 2024-04-15T22:28:42.367556+0000) 2024-04-15T22:28:44.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:44 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:44.267+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:45.176 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:44 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:44.907+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:45.177 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:45 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:45.131+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:45.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.82 MB/sec execute 54 sec latency 30773.673 ms 2024-04-15T22:28:45.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:45 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:45.312+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:45.750 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:45 smithi081 podman[58406]: 2024-04-15 22:28:45.462068033 +0000 UTC m=+5.684652577 container died d878f89983b510dd41a19059f80a1f89c77d734a03e7ded954f867b7583e1c31 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, ceph=True, distribution-scope=public, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T22:28:46.054 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:45 smithi081 podman[58406]: 2024-04-15 22:28:45.85079409 +0000 UTC m=+6.073378633 container cleanup d878f89983b510dd41a19059f80a1f89c77d734a03e7ded954f867b7583e1c31 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_BRANCH=HEAD, release=754, io.buildah.version=1.19.8, vcs-type=git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, version=8) 2024-04-15T22:28:46.054 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:45 smithi081 bash[58406]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3 2024-04-15T22:28:46.054 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:45 smithi081 bash[58438]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T22:28:46.176 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:45 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:45.893+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:46.177 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:46 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:46.087+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.71 MB/sec execute 55 sec latency 31773.719 ms 2024-04-15T22:28:46.446 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:46 smithi081 podman[58426]: 2024-04-15 22:28:46.102004132 +0000 UTC m=+0.637382333 container remove d878f89983b510dd41a19059f80a1f89c77d734a03e7ded954f867b7583e1c31 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3, name=centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-type=git, io.openshift.expose-services=, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, version=8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T22:28:46.447 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:46 smithi081 podman[58498]: 2024-04-15 22:28:46.212665615 +0000 UTC m=+0.018846900 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:28:46.447 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:46 smithi081 podman[58498]: 2024-04-15 22:28:46.319567602 +0000 UTC m=+0.125748881 container create 378542abcf9e11f593eb7002fa3f07f4073d098669259b5dc02b29e6fd45c532 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, distribution-scope=public, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vcs-type=git, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, release=754, architecture=x86_64, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:28:46.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:46 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:46.346+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:46.804 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:46 smithi081 podman[58498]: 2024-04-15 22:28:46.538811058 +0000 UTC m=+0.344992340 container init 378542abcf9e11f593eb7002fa3f07f4073d098669259b5dc02b29e6fd45c532 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux ) 2024-04-15T22:28:46.804 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:46 smithi081 podman[58498]: 2024-04-15 22:28:46.548679718 +0000 UTC m=+0.354861006 container start 378542abcf9e11f593eb7002fa3f07f4073d098669259b5dc02b29e6fd45c532 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-04-15T22:28:46.804 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:46 smithi081 podman[58498]: 2024-04-15 22:28:46.630501744 +0000 UTC m=+0.436683037 container attach 378542abcf9e11f593eb7002fa3f07f4073d098669259b5dc02b29e6fd45c532 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.openshift.tags=base centos centos-stream, vcs-type=git, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container) 2024-04-15T22:28:47.224 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 15 22:28:46 smithi081 podman[58498]: 2024-04-15 22:28:46.907112382 +0000 UTC m=+0.713293670 container died 378542abcf9e11f593eb7002fa3f07f4073d098669259b5dc02b29e6fd45c532 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , name=centos-stream, RELEASE=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., vcs-type=git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:28:47.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.61 MB/sec execute 56 sec latency 32773.778 ms 2024-04-15T22:28:47.309 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:47 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:47.034+0000 7fe1d10b7700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:20.264763+0000 front 2024-04-15T22:28:20.261022+0000 (oldest deadline 2024-04-15T22:28:46.161156+0000) 2024-04-15T22:28:47.310 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:46 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:46.939+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:47.310 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:47 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:47.044+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:47.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-15T22:28:47.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:47.603 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-15T22:28:47.604 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-15T22:28:47.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.4 2024-04-15T22:28:47.636 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:47 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:47.309+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:47.926 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:47 smithi163 systemd[1]: Stopping Ceph osd.4 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:47.927 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:47 smithi163 bash[43297]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T22:28:47.927 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:47 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:47.803+0000 7fe1d471c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:47.927 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:47 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:47.803+0000 7fe1d471c700 -1 osd.4 69 *** Got signal Terminated *** 2024-04-15T22:28:47.927 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:47 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:47.803+0000 7fe1d471c700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:28:48.278 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:47 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:47.945+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:48.278 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:48 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:48.031+0000 7fe1d10b7700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:20.264763+0000 front 2024-04-15T22:28:20.261022+0000 (oldest deadline 2024-04-15T22:28:46.161156+0000) 2024-04-15T22:28:48.279 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:48 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:48.092+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:48.301 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.51 MB/sec execute 57 sec latency 33773.842 ms 2024-04-15T22:28:48.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:48 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:48.278+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:49.068 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:48 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:48.931+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:49.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.42 MB/sec execute 58 sec latency 34773.906 ms 2024-04-15T22:28:49.426 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:49 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:49.309+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:49.427 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:49 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:49.068+0000 7fe1d10b7700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:20.264763+0000 front 2024-04-15T22:28:20.261022+0000 (oldest deadline 2024-04-15T22:28:46.161156+0000) 2024-04-15T22:28:49.428 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:49 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:49.142+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.32 MB/sec execute 59 sec latency 35773.973 ms 2024-04-15T22:28:50.307 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:49 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:49.968+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:50.307 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:50 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:50.043+0000 7fe1d10b7700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:20.264763+0000 front 2024-04-15T22:28:20.261022+0000 (oldest deadline 2024-04-15T22:28:46.161156+0000) 2024-04-15T22:28:50.308 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:50 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:50.096+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:50.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:50 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:50.307+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:50.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:50 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:50.307+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:51.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.24 MB/sec execute 60 sec latency 36774.012 ms 2024-04-15T22:28:51.328 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:51 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:51.016+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:51.329 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:51 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:51.016+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:51.329 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:51 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:51.026+0000 7fe1d10b7700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:20.264763+0000 front 2024-04-15T22:28:20.261022+0000 (oldest deadline 2024-04-15T22:28:46.161156+0000) 2024-04-15T22:28:51.329 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:51 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:51.106+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:51.329 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:51 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:51.106+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:51.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:51 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:51.329+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:51.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:51 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:51.329+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:52.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.15 MB/sec execute 61 sec latency 37774.044 ms 2024-04-15T22:28:52.335 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:52.053+0000 7fe1d10b7700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:20.264763+0000 front 2024-04-15T22:28:20.261022+0000 (oldest deadline 2024-04-15T22:28:46.161156+0000) 2024-04-15T22:28:52.336 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4[31298]: 2024-04-15T22:28:52.053+0000 7fe1d10b7700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.262177+0000 front 2024-04-15T22:28:27.267911+0000 (oldest deadline 2024-04-15T22:28:51.962417+0000) 2024-04-15T22:28:52.336 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:52.120+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:52.336 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:52.120+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:52.337 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:52.043+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:52.337 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:52.043+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:52.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:52.336+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:52.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:52 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:52.336+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:53.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 5.07 MB/sec execute 62 sec latency 38774.101 ms 2024-04-15T22:28:53.359 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:53.108+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:53.360 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:53.108+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:53.360 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:53.073+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:53.360 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:53.073+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:53.360 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:53.305+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:53.361 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:53 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:53.305+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:53.672 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:53 smithi163 podman[43309]: 2024-04-15 22:28:53.361006019 +0000 UTC m=+5.657322205 container died 4e994b835b7e11860fb60f059b30d356458e4a76fbb950ef5b146e9f8c6ff9c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4, name=centos-stream, ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_CLEAN=True, distribution-scope=public, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc.) 2024-04-15T22:28:53.926 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:53 smithi163 podman[43309]: 2024-04-15 22:28:53.838691879 +0000 UTC m=+6.135008097 container cleanup 4e994b835b7e11860fb60f059b30d356458e4a76fbb950ef5b146e9f8c6ff9c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4, maintainer=Guillaume Abrioux , version=8, architecture=x86_64, GIT_CLEAN=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, release=754, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git) 2024-04-15T22:28:53.926 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:53 smithi163 bash[43309]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4 2024-04-15T22:28:53.927 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:53 smithi163 bash[43342]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T22:28:54.288 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:54 smithi163 podman[43324]: 2024-04-15 22:28:54.139986138 +0000 UTC m=+0.772751764 container remove 4e994b835b7e11860fb60f059b30d356458e4a76fbb950ef5b146e9f8c6ff9c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, distribution-scope=public, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True) 2024-04-15T22:28:54.289 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:54 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:54.158+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:54.289 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:54 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:54.158+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:54.289 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:54 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:54.029+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:54.289 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:54 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:54.029+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.99 MB/sec execute 63 sec latency 39774.154 ms 2024-04-15T22:28:54.567 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:54 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:54.289+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:54.567 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:54 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:54.289+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:54.567 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:54 smithi163 podman[43406]: 2024-04-15 22:28:54.235824994 +0000 UTC m=+0.018852805 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:28:54.567 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:54 smithi163 podman[43406]: 2024-04-15 22:28:54.332575885 +0000 UTC m=+0.115603764 container create a2f9414d1eae555560b38c53578297262515043bd65c693c99f819a4554d5b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-type=git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, release=754, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:28:54.567 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:54 smithi163 podman[43406]: 2024-04-15 22:28:54.568072096 +0000 UTC m=+0.351099902 container init a2f9414d1eae555560b38c53578297262515043bd65c693c99f819a4554d5b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , version=8, RELEASE=HEAD, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-type=git, io.buildah.version=1.19.8, release=754, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-04-15T22:28:54.849 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:54 smithi163 podman[43406]: 2024-04-15 22:28:54.571680067 +0000 UTC m=+0.354707873 container start a2f9414d1eae555560b38c53578297262515043bd65c693c99f819a4554d5b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., distribution-scope=public, vcs-type=git, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, architecture=x86_64, RELEASE=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.component=centos-stream-container) 2024-04-15T22:28:54.850 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:54 smithi163 podman[43406]: 2024-04-15 22:28:54.693036191 +0000 UTC m=+0.476064027 container attach a2f9414d1eae555560b38c53578297262515043bd65c693c99f819a4554d5b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, version=8, GIT_CLEAN=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=) 2024-04-15T22:28:55.122 INFO:journalctl@ceph.osd.4.smithi163.stdout:Apr 15 22:28:54 smithi163 podman[43406]: 2024-04-15 22:28:54.851038443 +0000 UTC m=+0.634066258 container died a2f9414d1eae555560b38c53578297262515043bd65c693c99f819a4554d5b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, ceph=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T22:28:55.123 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:55.072+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:55.123 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:55.072+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:55.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.91 MB/sec execute 64 sec latency 40774.203 ms 2024-04-15T22:28:55.427 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:55.122+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:55.427 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:55.123+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:55.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:55.338+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:55.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:55.338+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:55.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-15T22:28:55.672 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:55.672 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-15T22:28:55.672 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-15T22:28:55.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.5 2024-04-15T22:28:55.831 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:55 smithi163 systemd[1]: Stopping Ceph osd.5 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:55.831 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:55 smithi163 bash[43486]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T22:28:56.083 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:56 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:56.036+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:56.083 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:56 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:56.036+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:56.083 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:55.934+0000 7f5c880c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:56.083 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:55.934+0000 7f5c880c3700 -1 osd.5 69 *** Got signal Terminated *** 2024-04-15T22:28:56.084 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:55 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:55.934+0000 7f5c880c3700 -1 osd.5 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:28:56.084 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:56 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:56.083+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:56.084 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:56 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:56.083+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:56.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.83 MB/sec execute 65 sec latency 41774.251 ms 2024-04-15T22:28:56.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:56 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:56.361+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:56.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:56 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:56.361+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:57.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.76 MB/sec execute 66 sec latency 42774.302 ms 2024-04-15T22:28:57.344 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:57 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:57.061+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:57.344 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:57 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:57.061+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:57.345 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:57 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:57.121+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:57.345 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:57 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:57.121+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:57.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:57 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:57.345+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:57.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:57 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:57.345+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:58.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.69 MB/sec execute 67 sec latency 43774.368 ms 2024-04-15T22:28:58.426 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:58 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:58.104+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:58.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:58 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:58.104+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:58.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:58 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:58.322+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:58.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:58 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:58.322+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:28:58.427 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:58 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:58.168+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:58.428 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:58 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:58.168+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:59.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.62 MB/sec execute 68 sec latency 44774.418 ms 2024-04-15T22:28:59.426 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:59 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:59.215+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:28:59.427 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:28:59 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:28:59.215+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:28:59.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:59 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:59.055+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:28:59.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:28:59 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:28:59.055+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:28:59.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:59 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:59.285+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:28:59.428 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:28:59 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:28:59.285+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:00.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.55 MB/sec execute 69 sec latency 45774.475 ms 2024-04-15T22:29:00.426 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:00 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:29:00.196+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:17.148904+0000 front 2024-04-15T22:28:17.149043+0000 (oldest deadline 2024-04-15T22:28:42.449045+0000) 2024-04-15T22:29:00.427 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:00 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:29:00.196+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.551001+0000 front 2024-04-15T22:28:27.557402+0000 (oldest deadline 2024-04-15T22:28:51.051151+0000) 2024-04-15T22:29:00.427 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:00 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5[34278]: 2024-04-15T22:29:00.196+0000 7f5c8325b700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.252571+0000 front 2024-04-15T22:28:36.252611+0000 (oldest deadline 2024-04-15T22:28:59.752655+0000) 2024-04-15T22:29:00.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:00 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:00.069+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:00.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:00 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:00.069+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:00.428 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:00 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:00.260+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:00.428 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:00 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:00.260+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:01.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.49 MB/sec execute 70 sec latency 46774.534 ms 2024-04-15T22:29:01.347 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:01.031+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:01.347 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:01.031+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:01.347 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:01.031+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:01.348 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:01.214+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:01.348 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:01.214+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:01.348 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:01.214+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:01.676 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:01 smithi163 podman[43499]: 2024-04-15 22:29:01.348341644 +0000 UTC m=+5.555244596 container died a25f44d5b8774491896848d79e6b246c9001e2c10f4425c1d696c041c66aaaf9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5, GIT_BRANCH=HEAD, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.expose-services=) 2024-04-15T22:29:01.982 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:01 smithi163 podman[43499]: 2024-04-15 22:29:01.724779496 +0000 UTC m=+5.931682439 container cleanup a25f44d5b8774491896848d79e6b246c9001e2c10f4425c1d696c041c66aaaf9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5, name=centos-stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, ceph=True, RELEASE=HEAD, vcs-type=git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T22:29:01.985 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:01 smithi163 bash[43499]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5 2024-04-15T22:29:01.986 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:01 smithi163 bash[43542]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T22:29:02.234 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:02 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:02.230+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:02.234 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:02 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:02.230+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:02.234 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:02 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:02.230+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:02.234 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:01.982+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:02.235 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:01.982+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:02.235 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:01 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:01.983+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:02.235 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:02 smithi163 podman[43529]: 2024-04-15 22:29:02.017889508 +0000 UTC m=+0.670272042 container remove a25f44d5b8774491896848d79e6b246c9001e2c10f4425c1d696c041c66aaaf9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, vcs-type=git, RELEASE=HEAD, name=centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T22:29:02.235 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:02 smithi163 podman[43608]: 2024-04-15 22:29:02.128751626 +0000 UTC m=+0.018944275 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:02.235 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:02 smithi163 podman[43608]: 2024-04-15 22:29:02.23528262 +0000 UTC m=+0.125475283 container create 9303438692f13ff02a83d92dc432089219136cad3435fb9c025c9694a7454eaa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, architecture=x86_64, io.buildah.version=1.19.8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.expose-services=, ceph=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, release=754) 2024-04-15T22:29:02.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.42 MB/sec execute 71 sec latency 47774.584 ms 2024-04-15T22:29:02.503 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:02 smithi163 podman[43608]: 2024-04-15 22:29:02.429379867 +0000 UTC m=+0.319572489 container init 9303438692f13ff02a83d92dc432089219136cad3435fb9c025c9694a7454eaa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, version=8, RELEASE=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, ceph=True, GIT_BRANCH=HEAD, name=centos-stream, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, release=754, vcs-type=git) 2024-04-15T22:29:02.503 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:02 smithi163 podman[43608]: 2024-04-15 22:29:02.432776593 +0000 UTC m=+0.322969218 container start 9303438692f13ff02a83d92dc432089219136cad3435fb9c025c9694a7454eaa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, distribution-scope=public, ceph=True, vcs-type=git, name=centos-stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True) 2024-04-15T22:29:02.503 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:02 smithi163 podman[43608]: 2024-04-15 22:29:02.504302094 +0000 UTC m=+0.394494730 container attach 9303438692f13ff02a83d92dc432089219136cad3435fb9c025c9694a7454eaa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, version=8, release=754, distribution-scope=public) 2024-04-15T22:29:02.927 INFO:journalctl@ceph.osd.5.smithi163.stdout:Apr 15 22:29:02 smithi163 podman[43608]: 2024-04-15 22:29:02.714359222 +0000 UTC m=+0.604551844 container died 9303438692f13ff02a83d92dc432089219136cad3435fb9c025c9694a7454eaa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.19.8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, release=754, version=8, maintainer=Guillaume Abrioux ) 2024-04-15T22:29:03.256 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:03.189+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:03.256 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:03.189+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:03.256 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:03.189+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:03.256 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:02 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:02.947+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:03.257 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:02 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:02.947+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:03.257 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:02 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:02.947+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:03.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-15T22:29:03.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.36 MB/sec execute 72 sec latency 48774.645 ms 2024-04-15T22:29:03.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:03.317 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-15T22:29:03.317 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-15T22:29:03.317 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.6 2024-04-15T22:29:03.534 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 systemd[1]: Stopping Ceph osd.6 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:03.535 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 bash[43687]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T22:29:03.926 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:03.535+0000 7fabbd343700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:03.926 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:03.535+0000 7fabbd343700 -1 osd.6 69 *** Got signal Terminated *** 2024-04-15T22:29:03.927 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:03.535+0000 7fabbd343700 -1 osd.6 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:04.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.30 MB/sec execute 73 sec latency 49774.694 ms 2024-04-15T22:29:04.426 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:03.947+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:04.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:03.947+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:04.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:03 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:03.947+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:04.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:04 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:04.188+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:04.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:04 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:04.188+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:04.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:04 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:04.188+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:05.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.25 MB/sec execute 74 sec latency 50774.758 ms 2024-04-15T22:29:05.426 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:04 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:04.964+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:05.426 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:04 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:04.964+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:05.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:04 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:04.964+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:05.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:05.184+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:05.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:05.184+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:05.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:05.184+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:06.247 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:05.977+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:06.247 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:05.977+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:06.247 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:05 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:05.977+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:06.247 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:06 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:06.193+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:06.247 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:06 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:06.193+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:06.248 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:06 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:06.193+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:06.248 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:06 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:06.193+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:06.302 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.19 MB/sec execute 75 sec latency 51774.815 ms 2024-04-15T22:29:07.170 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:06 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:06.952+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:07.171 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:06 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:06.952+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:07.171 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:06 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:06.952+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:07.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.13 MB/sec execute 76 sec latency 52774.892 ms 2024-04-15T22:29:07.426 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:07 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:07.171+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:07.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:07 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:07.171+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:07.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:07 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:07.171+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:07.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:07 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:07.171+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:08.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.08 MB/sec execute 77 sec latency 53774.964 ms 2024-04-15T22:29:08.426 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:07.999+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.909654+0000 front 2024-04-15T22:28:19.909664+0000 (oldest deadline 2024-04-15T22:28:43.409667+0000) 2024-04-15T22:29:08.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:07.999+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:27.411174+0000 front 2024-04-15T22:28:27.411185+0000 (oldest deadline 2024-04-15T22:28:50.911181+0000) 2024-04-15T22:29:08.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:07.999+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:36.812253+0000 front 2024-04-15T22:28:36.813813+0000 (oldest deadline 2024-04-15T22:29:00.312315+0000) 2024-04-15T22:29:08.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6[37334]: 2024-04-15T22:29:07.999+0000 7fabb64d7700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:42.613154+0000 front 2024-04-15T22:28:42.619364+0000 (oldest deadline 2024-04-15T22:29:07.313140+0000) 2024-04-15T22:29:08.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:08.160+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:08.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:08.160+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:08.428 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:08.160+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:08.428 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:08 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:08.160+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:09.176 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:08 smithi163 podman[43700]: 2024-04-15 22:29:08.900946128 +0000 UTC m=+5.469221890 container died c695e1b9459de00c7667f1b34fb1d5ff3554e99ea5d9be2f4222ed35e591a6a7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_BRANCH=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, distribution-scope=public, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True) 2024-04-15T22:29:09.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 4.03 MB/sec execute 78 sec latency 54775.012 ms 2024-04-15T22:29:09.426 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 podman[43700]: 2024-04-15 22:29:09.268235178 +0000 UTC m=+5.836510923 container cleanup c695e1b9459de00c7667f1b34fb1d5ff3554e99ea5d9be2f4222ed35e591a6a7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, ceph=True, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, release=754, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, distribution-scope=public) 2024-04-15T22:29:09.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 bash[43700]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6 2024-04-15T22:29:09.427 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 bash[43730]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T22:29:09.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:09 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:09.209+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:09.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:09 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:09.209+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:09.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:09 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:09.209+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:09.428 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:09 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:09.209+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:09.825 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 podman[43719]: 2024-04-15 22:29:09.527910344 +0000 UTC m=+0.622189281 container remove c695e1b9459de00c7667f1b34fb1d5ff3554e99ea5d9be2f4222ed35e591a6a7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, ceph=True, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, version=8, io.openshift.expose-services=, distribution-scope=public, maintainer=Guillaume Abrioux , release=754, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T22:29:09.825 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 podman[43796]: 2024-04-15 22:29:09.605247835 +0000 UTC m=+0.018798099 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:09.825 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 podman[43796]: 2024-04-15 22:29:09.72023653 +0000 UTC m=+0.133786839 container create ec4d8f39c811e9e0a9a92b8e8334865b4c56be618bf097febd6fb8033c3af58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, version=8, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_CLEAN=True, RELEASE=HEAD, vendor=Red Hat, Inc., distribution-scope=public, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:29:10.176 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 podman[43796]: 2024-04-15 22:29:09.955828251 +0000 UTC m=+0.369378514 container init ec4d8f39c811e9e0a9a92b8e8334865b4c56be618bf097febd6fb8033c3af58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, io.k8s.display-name=CentOS Stream 8, ceph=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, release=754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc.) 2024-04-15T22:29:10.177 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:09 smithi163 podman[43796]: 2024-04-15 22:29:09.95942027 +0000 UTC m=+0.372970538 container start ec4d8f39c811e9e0a9a92b8e8334865b4c56be618bf097febd6fb8033c3af58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.component=centos-stream-container, name=centos-stream, GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-type=git, distribution-scope=public) 2024-04-15T22:29:10.177 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:10 smithi163 podman[43796]: 2024-04-15 22:29:10.039107255 +0000 UTC m=+0.452657532 container attach ec4d8f39c811e9e0a9a92b8e8334865b4c56be618bf097febd6fb8033c3af58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.component=centos-stream-container, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:29:10.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.98 MB/sec execute 79 sec latency 55775.061 ms 2024-04-15T22:29:10.548 INFO:journalctl@ceph.osd.6.smithi163.stdout:Apr 15 22:29:10 smithi163 podman[43796]: 2024-04-15 22:29:10.257236775 +0000 UTC m=+0.670787042 container died ec4d8f39c811e9e0a9a92b8e8334865b4c56be618bf097febd6fb8033c3af58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, name=centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.component=centos-stream-container, version=8, io.buildah.version=1.19.8, ceph=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-type=git, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:29:10.549 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:10 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:10.230+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:10.549 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:10 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:10.230+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:10.549 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:10 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:10.230+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:10.549 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:10 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:10.230+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:10.858 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.6.service' 2024-04-15T22:29:10.900 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:10.900 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-15T22:29:10.901 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-15T22:29:10.901 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.7 2024-04-15T22:29:11.144 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:10 smithi163 systemd[1]: Stopping Ceph osd.7 for e7d16278-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:11.144 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 bash[43876]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T22:29:11.144 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:11.144+0000 7fcb1530f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:11.144 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:11.144+0000 7fcb1530f700 -1 osd.7 69 *** Got signal Terminated *** 2024-04-15T22:29:11.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.93 MB/sec execute 80 sec latency 56775.123 ms 2024-04-15T22:29:11.426 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:11.144+0000 7fcb1530f700 -1 osd.7 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:11.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:11.204+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:11.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:11.204+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:11.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:11.204+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:11.427 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:11 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:11.204+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:12.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.88 MB/sec execute 81 sec latency 57775.175 ms 2024-04-15T22:29:12.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:12 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:12.253+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:12.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:12 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:12.253+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:12.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:12 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:12.253+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:12.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:12 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:12.253+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:13.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.83 MB/sec execute 82 sec latency 58775.228 ms 2024-04-15T22:29:13.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:13 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:13.265+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:13.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:13 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:13.265+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:13.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:13 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:13.265+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:13.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:13 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:13.265+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:14.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.79 MB/sec execute 83 sec latency 59775.274 ms 2024-04-15T22:29:14.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:14 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:14.297+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:14.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:14 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:14.297+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:14.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:14 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:14.297+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:14.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:14 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:14.297+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:15.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.74 MB/sec execute 84 sec latency 60775.311 ms 2024-04-15T22:29:15.676 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:15 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:15.268+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6806 osd.0 since back 2024-04-15T22:28:19.352851+0000 front 2024-04-15T22:28:19.352753+0000 (oldest deadline 2024-04-15T22:28:41.652775+0000) 2024-04-15T22:29:15.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:15 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:15.268+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6814 osd.1 since back 2024-04-15T22:28:28.554812+0000 front 2024-04-15T22:28:28.554879+0000 (oldest deadline 2024-04-15T22:28:49.654855+0000) 2024-04-15T22:29:15.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:15 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:15.268+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6822 osd.2 since back 2024-04-15T22:28:31.355903+0000 front 2024-04-15T22:28:31.360447+0000 (oldest deadline 2024-04-15T22:29:00.756555+0000) 2024-04-15T22:29:15.677 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:15 smithi163 ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7[40346]: 2024-04-15T22:29:15.268+0000 7fcb12cac700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.81:6830 osd.3 since back 2024-04-15T22:28:43.558138+0000 front 2024-04-15T22:28:43.558129+0000 (oldest deadline 2024-04-15T22:29:05.858184+0000) 2024-04-15T22:29:16.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.70 MB/sec execute 85 sec latency 61775.357 ms 2024-04-15T22:29:17.001 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:16 smithi163 podman[43888]: 2024-04-15 22:29:16.723231379 +0000 UTC m=+5.700305356 container died 8da0636b871f295bd9766b4d2af959d70b172cb8e76eff1013c1f78a89fad976 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, version=8, io.openshift.expose-services=, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-04-15T22:29:17.001 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43888]: 2024-04-15 22:29:17.002985046 +0000 UTC m=+5.980059018 container cleanup 8da0636b871f295bd9766b4d2af959d70b172cb8e76eff1013c1f78a89fad976 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, RELEASE=HEAD, release=754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_CLEAN=True) 2024-04-15T22:29:17.252 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 bash[43888]: ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7 2024-04-15T22:29:17.253 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 bash[43919]: Error: no container with name or ID "ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T22:29:17.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.65 MB/sec execute 86 sec latency 62775.378 ms 2024-04-15T22:29:17.534 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43907]: 2024-04-15 22:29:17.253997727 +0000 UTC m=+0.530321944 container remove 8da0636b871f295bd9766b4d2af959d70b172cb8e76eff1013c1f78a89fad976 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-type=git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, ceph=True, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T22:29:17.535 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43987]: 2024-04-15 22:29:17.339538702 +0000 UTC m=+0.018670662 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:17.535 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43987]: 2024-04-15 22:29:17.446206827 +0000 UTC m=+0.125338776 container create 594a99e92fd36eb396ca51a43caadf627f9c79af67b3fb12cd96bb8e24adc0f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_BRANCH=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.display-name=CentOS Stream 8) 2024-04-15T22:29:17.921 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43987]: 2024-04-15 22:29:17.640102959 +0000 UTC m=+0.319234907 container init 594a99e92fd36eb396ca51a43caadf627f9c79af67b3fb12cd96bb8e24adc0f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, distribution-scope=public, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.component=centos-stream-container, name=centos-stream, ceph=True, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T22:29:17.921 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43987]: 2024-04-15 22:29:17.643458265 +0000 UTC m=+0.322590217 container start 594a99e92fd36eb396ca51a43caadf627f9c79af67b3fb12cd96bb8e24adc0f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-04-15T22:29:17.922 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43987]: 2024-04-15 22:29:17.731751754 +0000 UTC m=+0.410883714 container attach 594a99e92fd36eb396ca51a43caadf627f9c79af67b3fb12cd96bb8e24adc0f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, name=centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vendor=Red Hat, Inc.) 2024-04-15T22:29:17.922 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:17 smithi163 podman[43987]: 2024-04-15 22:29:17.922617137 +0000 UTC m=+0.601749076 container died 594a99e92fd36eb396ca51a43caadf627f9c79af67b3fb12cd96bb8e24adc0f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8) 2024-04-15T22:29:18.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.61 MB/sec execute 87 sec latency 63775.444 ms 2024-04-15T22:29:18.477 INFO:journalctl@ceph.osd.7.smithi163.stdout:Apr 15 22:29:18 smithi163 podman[44013]: 2024-04-15 22:29:18.459945363 +0000 UTC m=+0.531931929 container remove 594a99e92fd36eb396ca51a43caadf627f9c79af67b3fb12cd96bb8e24adc0f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e7d16278-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, version=8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, ceph=True, GIT_BRANCH=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, maintainer=Guillaume Abrioux , release=754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, RELEASE=HEAD) 2024-04-15T22:29:18.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e7d16278-fb75-11ee-bc8f-c7b262605968@osd.7.service' 2024-04-15T22:29:18.524 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:18.524 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-15T22:29:18.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T22:29:19.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.57 MB/sec execute 88 sec latency 64775.490 ms 2024-04-15T22:29:20.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.53 MB/sec execute 89 sec latency 65775.540 ms 2024-04-15T22:29:21.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.49 MB/sec execute 90 sec latency 66775.610 ms 2024-04-15T22:29:22.303 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.45 MB/sec execute 91 sec latency 67775.673 ms 2024-04-15T22:29:23.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.41 MB/sec execute 92 sec latency 68775.711 ms 2024-04-15T22:29:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.38 MB/sec execute 93 sec latency 69775.741 ms 2024-04-15T22:29:25.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.34 MB/sec execute 94 sec latency 70775.792 ms 2024-04-15T22:29:26.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.31 MB/sec execute 95 sec latency 71775.843 ms 2024-04-15T22:29:27.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.27 MB/sec execute 96 sec latency 72775.906 ms 2024-04-15T22:29:28.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.24 MB/sec execute 97 sec latency 73775.972 ms 2024-04-15T22:29:29.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.21 MB/sec execute 98 sec latency 74775.988 ms 2024-04-15T22:29:30.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.17 MB/sec execute 99 sec latency 75776.021 ms 2024-04-15T22:29:31.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.14 MB/sec execute 100 sec latency 76776.073 ms 2024-04-15T22:29:32.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.11 MB/sec execute 101 sec latency 77776.120 ms 2024-04-15T22:29:33.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.08 MB/sec execute 102 sec latency 78776.169 ms 2024-04-15T22:29:34.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.05 MB/sec execute 103 sec latency 79776.189 ms 2024-04-15T22:29:35.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 3.02 MB/sec execute 104 sec latency 80776.229 ms 2024-04-15T22:29:36.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.99 MB/sec execute 105 sec latency 81776.276 ms 2024-04-15T22:29:37.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.96 MB/sec execute 106 sec latency 82776.320 ms 2024-04-15T22:29:38.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.94 MB/sec execute 107 sec latency 83776.353 ms 2024-04-15T22:29:39.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.91 MB/sec execute 108 sec latency 84776.401 ms 2024-04-15T22:29:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.88 MB/sec execute 109 sec latency 85776.449 ms 2024-04-15T22:29:41.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.86 MB/sec execute 110 sec latency 86776.483 ms 2024-04-15T22:29:42.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.83 MB/sec execute 111 sec latency 87776.522 ms 2024-04-15T22:29:43.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.81 MB/sec execute 112 sec latency 88776.555 ms 2024-04-15T22:29:44.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.78 MB/sec execute 113 sec latency 89776.590 ms 2024-04-15T22:29:45.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.76 MB/sec execute 114 sec latency 90776.633 ms 2024-04-15T22:29:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.73 MB/sec execute 115 sec latency 91776.661 ms 2024-04-15T22:29:46.792 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T22:29:47.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.71 MB/sec execute 116 sec latency 92776.704 ms 2024-04-15T22:29:48.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.69 MB/sec execute 117 sec latency 93776.730 ms 2024-04-15T22:29:49.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.66 MB/sec execute 118 sec latency 94776.825 ms 2024-04-15T22:29:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.64 MB/sec execute 119 sec latency 95776.908 ms 2024-04-15T22:29:51.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.62 MB/sec execute 120 sec latency 96776.975 ms 2024-04-15T22:29:52.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.60 MB/sec execute 121 sec latency 97777.077 ms 2024-04-15T22:29:53.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.58 MB/sec execute 122 sec latency 98777.150 ms 2024-04-15T22:29:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.55 MB/sec execute 123 sec latency 99777.249 ms 2024-04-15T22:29:55.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.53 MB/sec execute 124 sec latency 100777.340 ms 2024-04-15T22:29:56.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.51 MB/sec execute 125 sec latency 101777.382 ms 2024-04-15T22:29:57.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.49 MB/sec execute 126 sec latency 102777.488 ms 2024-04-15T22:29:58.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.47 MB/sec execute 127 sec latency 103777.607 ms 2024-04-15T22:29:59.305 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.45 MB/sec execute 128 sec latency 104777.687 ms 2024-04-15T22:30:00.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.44 MB/sec execute 129 sec latency 105777.778 ms 2024-04-15T22:30:01.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.42 MB/sec execute 130 sec latency 106777.855 ms 2024-04-15T22:30:02.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.40 MB/sec execute 131 sec latency 107777.974 ms 2024-04-15T22:30:03.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.38 MB/sec execute 132 sec latency 108778.079 ms 2024-04-15T22:30:04.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.36 MB/sec execute 133 sec latency 109778.136 ms 2024-04-15T22:30:05.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.34 MB/sec execute 134 sec latency 110778.229 ms 2024-04-15T22:30:06.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.33 MB/sec execute 135 sec latency 111778.309 ms 2024-04-15T22:30:07.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:30:07.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:30:07.203 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-15T22:30:07.205 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536/remote/smithi081/crash 2024-04-15T22:30:07.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T22:30:07.245 INFO:teuthology.orchestra.run.smithi081.stderr:tar: /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T22:30:07.245 INFO:teuthology.orchestra.run.smithi081.stderr:tar: Error is not recoverable: exiting now 2024-04-15T22:30:07.247 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536/remote/smithi163/crash 2024-04-15T22:30:07.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T22:30:07.287 INFO:teuthology.orchestra.run.smithi163.stderr:tar: /var/lib/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T22:30:07.288 INFO:teuthology.orchestra.run.smithi163.stderr:tar: Error is not recoverable: exiting now 2024-04-15T22:30:07.289 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-15T22:30:07.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-15T22:30:07.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.31 MB/sec execute 136 sec latency 112778.347 ms 2024-04-15T22:30:07.321 INFO:teuthology.orchestra.run.smithi081.stdout:2024-04-15T22:26:18.149603+0000 mon.smithi081 (mon.0) 760 : cluster [WRN] Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:30:07.321 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-15T22:30:07.321 INFO:tasks.cephadm:Compressing logs... 2024-04-15T22:30:07.322 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T22:30:07.363 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T22:30:07.396 INFO:teuthology.orchestra.run.smithi081.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-15T22:30:07.397 INFO:teuthology.orchestra.run.smithi081.stderr:: No such file or directory 2024-04-15T22:30:07.397 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mds.foofs.smithi081.qxeoot.log 2024-04-15T22:30:07.397 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mon.smithi081.log 2024-04-15T22:30:07.397 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mds.foofs.smithi081.qxeoot.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr: 82.0% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mds.foofs.smithi081.qxeoot.log.gz 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mon.smithi081.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mgr.smithi081.wfvsja.log 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mgr.smithi081.wfvsja.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.0.log: /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log: 90.8% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T22:30:07.398 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T22:30:07.399 INFO:teuthology.orchestra.run.smithi081.stderr: 83.0% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T22:30:07.399 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-15T22:30:07.399 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T22:30:07.399 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.2.log: 86.8% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T22:30:07.406 INFO:teuthology.orchestra.run.smithi163.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-15T22:30:07.406 INFO:teuthology.orchestra.run.smithi163.stderr:: No such file or directory 2024-04-15T22:30:07.407 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.6.log 2024-04-15T22:30:07.407 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mds.foofs.smithi163.xhjdei.log 2024-04-15T22:30:07.407 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mgr.smithi163.cdbjip.log 2024-04-15T22:30:07.407 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mds.foofs.smithi163.xhjdei.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T22:30:07.408 INFO:teuthology.orchestra.run.smithi163.stderr: 60.7% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mds.foofs.smithi163.xhjdei.log.gz 2024-04-15T22:30:07.408 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mgr.smithi163.cdbjip.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-15T22:30:07.408 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.7.log 2024-04-15T22:30:07.408 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-15T22:30:07.409 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T22:30:07.409 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T22:30:07.409 INFO:teuthology.orchestra.run.smithi163.stderr: 91.0% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T22:30:07.410 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log: 82.2% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T22:30:07.410 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T22:30:07.410 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mon.smithi163.log 2024-04-15T22:30:07.410 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.log: /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-volume.log: 86.8% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T22:30:07.411 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mon.smithi163.log: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T22:30:07.411 INFO:teuthology.orchestra.run.smithi163.stderr: 92.4% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mgr.smithi163.cdbjip.log.gz 2024-04-15T22:30:07.411 INFO:teuthology.orchestra.run.smithi163.stderr: 91.0% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T22:30:07.412 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-volume.log: 91.1% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T22:30:07.413 INFO:teuthology.orchestra.run.smithi081.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T22:30:07.550 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mgr.smithi081.wfvsja.log.gz 2024-04-15T22:30:07.786 INFO:teuthology.orchestra.run.smithi163.stderr: 93.5% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mon.smithi163.log.gz 2024-04-15T22:30:07.824 INFO:teuthology.orchestra.run.smithi081.stderr: 90.6% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-mon.smithi081.log.gz 2024-04-15T22:30:08.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.29 MB/sec execute 137 sec latency 113778.382 ms 2024-04-15T22:30:09.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.28 MB/sec execute 138 sec latency 114778.421 ms 2024-04-15T22:30:10.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.26 MB/sec execute 139 sec latency 115778.462 ms 2024-04-15T22:30:10.770 INFO:teuthology.orchestra.run.smithi081.stderr: 94.6% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-15T22:30:11.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.24 MB/sec execute 140 sec latency 116778.502 ms 2024-04-15T22:30:11.603 INFO:teuthology.orchestra.run.smithi163.stderr: 94.1% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-15T22:30:11.689 INFO:teuthology.orchestra.run.smithi163.stderr: 94.3% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.6.log.gz 2024-04-15T22:30:12.186 INFO:teuthology.orchestra.run.smithi081.stderr: 94.2% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-15T22:30:12.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.23 MB/sec execute 141 sec latency 117778.563 ms 2024-04-15T22:30:12.340 INFO:teuthology.orchestra.run.smithi081.stderr: 94.3% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-15T22:30:12.605 INFO:teuthology.orchestra.run.smithi081.stderr: 94.2% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-15T22:30:12.607 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-15T22:30:12.608 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m5.231s 2024-04-15T22:30:12.608 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m18.227s 2024-04-15T22:30:12.608 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m0.720s 2024-04-15T22:30:13.130 INFO:teuthology.orchestra.run.smithi163.stderr: 94.2% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-15T22:30:13.306 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.21 MB/sec execute 142 sec latency 118778.658 ms 2024-04-15T22:30:14.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.20 MB/sec execute 143 sec latency 119778.733 ms 2024-04-15T22:30:14.438 INFO:teuthology.orchestra.run.smithi163.stderr: 94.2% -- replaced with /var/log/ceph/e7d16278-fb75-11ee-bc8f-c7b262605968/ceph-osd.7.log.gz 2024-04-15T22:30:14.440 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-04-15T22:30:14.440 INFO:teuthology.orchestra.run.smithi163.stderr:real 0m7.061s 2024-04-15T22:30:14.440 INFO:teuthology.orchestra.run.smithi163.stderr:user 0m20.877s 2024-04-15T22:30:14.440 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m0.812s 2024-04-15T22:30:14.440 INFO:tasks.cephadm:Archiving logs... 2024-04-15T22:30:14.441 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536/remote/smithi081/log 2024-04-15T22:30:14.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T22:30:15.323 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.18 MB/sec execute 144 sec latency 120778.777 ms 2024-04-15T22:30:16.326 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.17 MB/sec execute 145 sec latency 121778.810 ms 2024-04-15T22:30:17.020 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536/remote/smithi163/log 2024-04-15T22:30:17.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T22:30:17.325 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.15 MB/sec execute 146 sec latency 122778.884 ms 2024-04-15T22:30:18.320 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.14 MB/sec execute 147 sec latency 123778.983 ms 2024-04-15T22:30:19.310 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.12 MB/sec execute 148 sec latency 124779.091 ms 2024-04-15T22:30:19.979 INFO:tasks.cephadm:Removing cluster... 2024-04-15T22:30:19.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 --force 2024-04-15T22:30:20.287 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 --force 2024-04-15T22:30:20.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.11 MB/sec execute 149 sec latency 125779.177 ms 2024-04-15T22:30:20.589 INFO:tasks.cephadm:Removing cephadm ... 2024-04-15T22:30:20.589 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T22:30:20.605 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T22:30:20.621 INFO:tasks.cephadm:Teardown complete 2024-04-15T22:30:20.622 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T22:30:20.633 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T22:30:20.633 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:30:20.648 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:30:20.660 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:30:20.665 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:30:20.665 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-04-15T22:30:20.665 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +924us[ +904us] +/- 47ms 2024-04-15T22:30:20.666 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -467us[ -487us] +/- 57ms 2024-04-15T22:30:20.666 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -918us[ -918us] +/- 71ms 2024-04-15T22:30:20.666 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:30:20.676 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:30:20.681 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:30:20.681 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-04-15T22:30:20.681 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv01.front.sepia.ceph.com 3 6 377 7 +918us[ +901us] +/- 47ms 2024-04-15T22:30:20.681 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -471us[ -471us] +/- 57ms 2024-04-15T22:30:20.681 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 -912us[ -912us] +/- 71ms 2024-04-15T22:30:20.681 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:30:20.682 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T22:30:20.692 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T22:30:20.693 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T22:30:20.702 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T22:30:20.782 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T22:30:20.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T22:30:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:30:20.887 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T22:30:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:30:20.919 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T22:30:20.928 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T22:30:20.937 INFO:teuthology.task.internal:Duration was 1063.585966 seconds 2024-04-15T22:30:20.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T22:30:20.947 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T22:30:20.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T22:30:20.950 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T22:30:20.994 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:30:21.004 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:30:21.310 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.09 MB/sec execute 150 sec latency 126779.231 ms 2024-04-15T22:30:21.357 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T22:30:21.357 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-04-15T22:30:21.358 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T22:30:21.381 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-04-15T22:30:21.381 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T22:30:21.412 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T22:30:21.412 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T22:30:21.424 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T22:30:21.483 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T22:30:21.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T22:30:21.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T22:30:22.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.08 MB/sec execute 151 sec latency 127779.258 ms 2024-04-15T22:30:23.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.07 MB/sec execute 152 sec latency 128779.287 ms 2024-04-15T22:30:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.05 MB/sec execute 153 sec latency 129779.329 ms 2024-04-15T22:30:25.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.04 MB/sec execute 154 sec latency 130779.358 ms 2024-04-15T22:30:26.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.03 MB/sec execute 155 sec latency 131779.388 ms 2024-04-15T22:30:26.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T22:30:26.699 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T22:30:26.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T22:30:26.728 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T22:30:26.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T22:30:26.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:30:26.775 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:30:26.800 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-04-15T22:30:26.834 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-04-15T22:30:26.863 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:30:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:30:26.877 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:30:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:30:26.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T22:30:26.931 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T22:30:26.932 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536/remote/smithi081 2024-04-15T22:30:26.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T22:30:26.985 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536/remote/smithi163 2024-04-15T22:30:26.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T22:30:27.027 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T22:30:27.028 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T22:30:27.030 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T22:30:27.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T22:30:27.091 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T22:30:27.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T22:30:27.103 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T22:30:27.104 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T22:30:27.106 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T22:30:27.119 INFO:teuthology.orchestra.run.smithi081.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 22:30 /home/ubuntu/cephtest 2024-04-15T22:30:27.136 INFO:teuthology.orchestra.run.smithi163.stdout: 265610 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 22:30 /home/ubuntu/cephtest 2024-04-15T22:30:27.137 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T22:30:27.148 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T22:30:27.169 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T22:30:27.197 DEBUG:teuthology.nuke:shortname: smithi081 2024-04-15T22:30:27.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T22:30:27.224 DEBUG:teuthology.nuke:shortname: smithi163 2024-04-15T22:30:27.224 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T22:30:27.228 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:14.352992', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE14Iq7Kjy9N+JtU1nVMkjYzi9Mp+oObvzqyMXMm5pAF483WD5vkoPI+bKtF6+56igBtQv1YEAqD7elEdt/dT9g='} 2024-04-15T22:30:27.248 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:14.351911', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEQSrcVKz7rr8Aj1XdGsSz8z5fvtonxc+F+aC6iCDktwfN92oYzaGY7eoEwFhmoBG6OAeJiOuo3WnRMEMktgrU='} 2024-04-15T22:30:27.285 INFO:teuthology.orchestra.console.smithi081:Power off 2024-04-15T22:30:27.285 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T22:30:27.307 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-04-15T22:30:27.307 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:30:27.317 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.01 MB/sec execute 156 sec latency 132779.469 ms 2024-04-15T22:30:27.325 INFO:teuthology.orchestra.console.smithi163:Power off 2024-04-15T22:30:27.325 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T22:30:27.335 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-04-15T22:30:27.348 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-04-15T22:30:27.348 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:30:27.371 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-04-15T22:30:28.308 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 2.00 MB/sec execute 157 sec latency 133779.542 ms 2024-04-15T22:30:29.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 1.99 MB/sec execute 158 sec latency 134779.630 ms 2024-04-15T22:30:30.307 INFO:teuthology.orchestra.run.smithi081.stdout: 5 90108 1.98 MB/sec execute 159 sec latency 135779.694 ms 2024-04-15T22:30:30.387 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:30:31.336 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:30:31.372 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:30:31.448 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-04-15T22:30:31.483 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-04-15T22:30:35.450 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:30:35.485 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:30:35.561 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-04-15T22:30:35.594 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-04-15T22:30:35.662 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-04-15T22:30:35.695 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-04-15T22:30:35.801 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 1063.5859656333923 failure_reason: 'Command failed on smithi081 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e7d16278-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | keys\''"\''"\'' | grep $sha1\''''' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ec3c4ddbc6724492904164c0f13ec73b status: fail success: false 2024-04-15T22:30:35.801 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T22:30:35.892 INFO:teuthology.run:FAIL