2024-03-30T14:06:59.875 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T14:06:59.875 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --archive /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164 --owner scheduled_teuthology@teuthology --name teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164/orig.config.yaml 2024-03-30T14:06:59.912 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T14:06:59.996 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7630164' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 5af0400050a123dc31f105425174f106e057276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 8777 sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb sleep_before_teardown: 0 subset: 43/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 5af0400050a123dc31f105425174f106e057276a targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG89GpAQgzBB1UWS2gugrrqHdDANXgdwJyovs2hbGhOMFTjTrrVd0H68lzVS58IpenXBpmPyqrIWz5Y/lL0SbZw= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLiVOeIf5DBouL85wfOi0IeP7IQgbrFG+2MJnAbKpWXvILGURQlJANt8hpdFnadcB9cLlPE4hA5hiW4ZRKLT8/8= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_21:08:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-03-30T14:06:59.996 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa; will attempt to use it 2024-03-30T14:06:59.997 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa/tasks 2024-03-30T14:06:59.997 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T14:06:59.999 INFO:teuthology.task.internal:Checking packages... 2024-03-30T14:07:00.023 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb' 2024-03-30T14:07:00.023 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T14:07:00.023 INFO:teuthology.packaging:ref: None 2024-03-30T14:07:00.023 INFO:teuthology.packaging:tag: None 2024-03-30T14:07:00.023 INFO:teuthology.packaging:branch: squid 2024-03-30T14:07:00.023 INFO:teuthology.packaging:sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:07:00.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-30T14:07:00.200 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1973.g5af04000 2024-03-30T14:07:00.202 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T14:07:00.208 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T14:07:00.208 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T14:07:00.215 INFO:teuthology.task.internal:Saving configuration 2024-03-30T14:07:00.229 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T14:07:00.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:07:00.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:02:15.051617', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG89GpAQgzBB1UWS2gugrrqHdDANXgdwJyovs2hbGhOMFTjTrrVd0H68lzVS58IpenXBpmPyqrIWz5Y/lL0SbZw='} 2024-03-30T14:07:00.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:02:15.053156', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLiVOeIf5DBouL85wfOi0IeP7IQgbrFG+2MJnAbKpWXvILGURQlJANt8hpdFnadcB9cLlPE4hA5hiW4ZRKLT8/8='} 2024-03-30T14:07:00.281 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T14:07:00.287 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-30T14:07:00.287 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-30T14:07:00.287 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T14:07:00.357 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2155b0ff70>, signals=[15]) 2024-03-30T14:07:00.357 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T14:07:00.363 INFO:teuthology.task.internal:Opening connections... 2024-03-30T14:07:00.364 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:07:00.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:07:00.472 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-03-30T14:07:00.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:07:00.561 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T14:07:00.568 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-03-30T14:07:00.586 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-03-30T14:07:00.587 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-03-30T14:07:00.643 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-03-30T14:07:00.643 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-03-30T14:07:00.643 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-03-30T14:07:00.643 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-03-30T14:07:00.643 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-03-30T14:07:00.643 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T14:07:00.644 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T14:07:00.645 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-03-30T14:07:00.669 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-03-30T14:07:00.686 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-03-30T14:07:00.687 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-03-30T14:07:00.742 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T14:07:00.743 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T14:07:00.744 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-03-30T14:07:00.761 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T14:07:00.771 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T14:07:00.776 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T14:07:00.776 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-03-30T14:07:00.779 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-03-30T14:07:00.798 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T14:07:00.803 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T14:07:00.804 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-03-30T14:07:00.836 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-03-30T14:07:00.862 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T14:07:00.917 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T14:07:00.932 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T14:07:00.932 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T14:07:00.932 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T14:07:00.933 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T14:07:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:07:00.950 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-30T14:07:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:07:00.951 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-03-30T14:07:01.007 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-361.el9.x86_64 2024-03-30T14:07:01.008 INFO:teuthology.task.kernel:Running kernel on smithi193: 5.14.0-361.el9.x86_64 2024-03-30T14:07:01.008 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2024-03-30T14:07:01.009 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-361.el9.x86_64 2024-03-30T14:07:01.009 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-361.el9.x86_64 2024-03-30T14:07:01.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-03-30T14:07:03.140 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-30T14:07:03.170 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-30T14:07:04.606 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-30T14:07:04.854 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-30T14:07:05.830 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-30T14:07:08.079 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-30T14:07:12.763 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-30T14:07:13.904 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 919 kB/s | 8.0 MB 00:08 2024-03-30T14:07:14.796 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 16 kB 00:00 2024-03-30T14:07:15.840 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-30T14:07:16.963 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 19 MB 00:01 2024-03-30T14:07:21.014 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-30T14:07:21.047 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-30T14:07:22.439 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-30T14:07:23.213 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T14:07:23.213 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T14:07:23.304 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T14:07:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:07:23.317 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-30T14:07:23.317 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-30T14:07:23.317 INFO:teuthology.orchestra.run.smithi193.stdout:Install 4 Packages 2024-03-30T14:07:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:07:23.317 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 76 M 2024-03-30T14:07:23.317 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 123 M 2024-03-30T14:07:23.317 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-30T14:07:23.482 INFO:teuthology.orchestra.run.smithi193.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-30T14:07:23.566 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-30T14:07:24.024 INFO:teuthology.orchestra.run.smithi193.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-30T14:07:24.649 INFO:teuthology.orchestra.run.smithi193.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-03-30T14:07:24.891 INFO:teuthology.orchestra.run.smithi193.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-30T14:07:24.894 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:07:24.894 INFO:teuthology.orchestra.run.smithi193.stdout:Total 48 MB/s | 76 MB 00:01 2024-03-30T14:07:24.895 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-30T14:07:25.253 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-30T14:07:25.254 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-30T14:07:26.081 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-30T14:07:26.081 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-30T14:07:28.390 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-30T14:07:28.832 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-03-30T14:07:28.982 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:07:29.074 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:07:31.173 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T14:07:31.173 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T14:07:31.279 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-30T14:07:31.289 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-30T14:07:31.289 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-30T14:07:31.289 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-30T14:07:31.289 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-30T14:07:31.289 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T14:07:31.289 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-03-30T14:07:31.290 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:07:31.291 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 76 M 2024-03-30T14:07:31.291 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 123 M 2024-03-30T14:07:31.291 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-30T14:07:31.542 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 764 kB/s | 192 kB 00:00 2024-03-30T14:07:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:07:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:07:32.367 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-30T14:07:32.667 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-30T14:07:33.025 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-30T14:07:33.028 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:07:33.029 INFO:teuthology.orchestra.run.smithi031.stdout:Total 44 MB/s | 76 MB 00:01 2024-03-30T14:07:33.029 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-30T14:07:33.453 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-30T14:07:33.453 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-30T14:07:34.319 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-30T14:07:34.319 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-30T14:07:36.709 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:07:36.750 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:07:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-30T14:07:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:07:37.576 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:07:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:07:40.618 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:07:41.532 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:07:45.687 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:07:45.725 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:07:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:10.814 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:10.814 INFO:teuthology.orchestra.run.smithi193.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T14:08:10.815 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:08:10.858 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:11.257 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:11.257 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:08:11.257 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:08:11.257 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:08:11.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:11.658 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:08:11.658 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-30T14:08:11.658 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T14:08:11.658 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T14:08:11.658 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T14:08:11.658 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T14:08:11.659 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:08:11.659 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-30T14:08:12.006 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:08:12.118 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:08:12.119 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:08:12.119 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T14:08:12.119 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T14:08:12.119 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T14:08:12.119 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-30T14:08:12.119 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T14:08:12.119 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:08:12.199 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:08:12.200 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:08:12.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.06 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 68.el9 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Group : Unspecified 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 8264254 2024-03-30T14:08:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : builder@centos.org 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : CentOS 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:08:12.235 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T14:08:12.236 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T14:08:12.236 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 5.14.0-434.el9.x86_64 2024-03-30T14:08:12.236 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T14:08:12.863 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2024-03-30T14:08:14.172 INFO:teuthology.orchestra.run.smithi193.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T14:08:14.205 INFO:teuthology.orchestra.run.smithi193.stderr:done 2024-03-30T14:08:14.208 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-03-30T14:08:14.223 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.BtupSwXbat 2024-03-30T14:08:14.223 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BtupSwXbat 2024-03-30T14:08:14.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.BtupSwXbat 2024-03-30T14:08:14.432 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.BtupSwXbat is 6KB 2024-03-30T14:08:14.481 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.BtupSwXbat 2024-03-30T14:08:14.495 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T14:08:14.566 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T14:08:14.566 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T14:08:14.566 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T14:08:14.566 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T14:08:14.568 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T14:08:14.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2024-03-30T14:08:21.593 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:21.594 INFO:teuthology.orchestra.run.smithi031.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T14:08:21.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:08:21.638 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:22.019 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:22.019 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:08:22.019 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:08:22.019 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:08:22.443 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-30T14:08:22.772 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:08:22.916 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:08:22.917 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:08:22.917 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T14:08:22.917 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T14:08:22.917 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T14:08:22.917 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-30T14:08:22.917 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T14:08:22.917 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:08:23.000 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:08:23.000 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:08:23.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 68.el9 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8264254 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-03-30T14:08:23.038 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-03-30T14:08:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:08:23.040 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T14:08:23.040 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T14:08:23.040 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-434.el9.x86_64 2024-03-30T14:08:23.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T14:08:23.663 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-03-30T14:08:25.023 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T14:08:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-03-30T14:08:25.062 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-03-30T14:08:25.077 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.3dEGUpzK8C 2024-03-30T14:08:25.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3dEGUpzK8C 2024-03-30T14:08:25.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.3dEGUpzK8C 2024-03-30T14:08:25.304 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.3dEGUpzK8C is 6KB 2024-03-30T14:08:25.325 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.3dEGUpzK8C 2024-03-30T14:08:25.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T14:08:25.409 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T14:08:25.409 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T14:08:25.409 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T14:08:25.409 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T14:08:25.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T14:08:25.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-03-30T14:08:44.727 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-30T14:08:44.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T14:08:44.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:08:55.582 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-30T14:08:55.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-30T14:08:55.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:09:03.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2024-03-30T14:09:12.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T14:09:12.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:09:14.018 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-30T14:09:23.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-30T14:09:23.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:09:54.809 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-03-30T14:09:55.197 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-30T14:09:55.197 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-434.el9.x86_64"... 2024-03-30T14:09:55.197 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-03-30T14:09:55.214 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-434.el9.x86_64 2024-03-30T14:09:55.215 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T14:09:55.215 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T14:09:55.215 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T14:09:56.215 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-30T14:09:56.216 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T14:09:56.305 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-03-30T14:09:56.379 DEBUG:teuthology.parallel:result is None 2024-03-30T14:10:12.273 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T14:10:24.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T14:10:24.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:10:24.722 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-30T14:10:25.089 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T14:10:25.089 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-434.el9.x86_64"... 2024-03-30T14:10:25.089 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-30T14:10:25.104 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-434.el9.x86_64 2024-03-30T14:10:25.104 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T14:10:25.104 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T14:10:25.105 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T14:10:26.105 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-30T14:10:26.106 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T14:10:26.456 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2024-03-30T14:10:26.488 DEBUG:teuthology.parallel:result is None 2024-03-30T14:10:26.488 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T14:10:26.497 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T14:10:26.497 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T14:10:26.501 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T14:10:26.518 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T14:10:26.530 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T14:10:26.538 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T14:10:26.538 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T14:10:26.561 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T14:10:26.590 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T14:10:26.598 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T14:10:26.598 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T14:10:26.628 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T14:10:26.670 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:10:26.673 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:10:26.684 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:10:26.687 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:10:26.688 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T14:10:26.697 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T14:10:26.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T14:10:26.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T14:10:26.729 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T14:10:26.739 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T14:10:26.740 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T14:10:26.769 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T14:10:26.785 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:10:26.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:10:26.918 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:10:26.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:10:27.042 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:10:27.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T14:10:27.109 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:10:27.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:10:27.205 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:10:27.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:10:27.332 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T14:10:27.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T14:10:27.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-03-30T14:10:27.403 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-03-30T14:10:27.447 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:10:27.484 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:10:27.895 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T14:10:27.906 INFO:teuthology.task.internal:Starting timer... 2024-03-30T14:10:27.906 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T14:10:27.923 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T14:10:27.933 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-30T14:10:27.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-03-30T14:10:27.989 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-03-30T14:10:27.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-03-30T14:10:28.058 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-03-30T14:10:28.060 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T14:10:28.060 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-03-30T14:10:28.087 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-03-30T14:10:28.087 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-03-30T14:10:28.119 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-03-30T14:10:28.119 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-03-30T14:10:28.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-30T14:10:28.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:10:28.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-30T14:10:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:10:28.187 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T14:10:28.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-03-30T14:10:28.231 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-03-30T14:10:28.261 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T14:10:28.272 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T14:10:28.287 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T14:10:28.288 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T14:14:45.304 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-03-30T14:14:45.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-30T14:14:45.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:14:45.396 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-03-30T14:14:45.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-30T14:14:45.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T14:14:45.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:14:45.562 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-30T14:14:45.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T14:14:45.642 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T14:14:45.653 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T14:14:45.653 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T14:14:45.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:14:45.656 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T14:14:45.656 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:14:45.685 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T14:14:45.701 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T14:14:45.712 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T14:14:45.728 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T14:14:45.731 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-03-30T14:14:45.744 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-03-30T14:14:45.758 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-03-30T14:14:45.761 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T14:14:45.771 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-03-30T14:14:45.777 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T14:14:45.788 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T14:14:45.804 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T14:14:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:14:45.834 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:14:45.834 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-03-30T14:14:45.834 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.834 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.834 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.834 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.856 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:14:45.859 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:14:45.859 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-30T14:14:45.859 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.859 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.859 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.859 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:14:45.860 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-30T14:14:45.956 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'} 2024-03-30T14:14:45.957 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-30T14:14:45.957 INFO:tasks.cephadm:Cluster fsid is dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:14:45.957 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-30T14:14:45.957 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-30T14:14:45.957 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi031': '172.21.15.31', 'mon.smithi193': '172.21.15.193'} 2024-03-30T14:14:45.958 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-30T14:14:45.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-03-30T14:14:45.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-03-30T14:14:46.013 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-30T14:14:46.014 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T14:14:46.244 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 30 14:14 /home/ubuntu/cephtest/cephadm 2024-03-30T14:14:46.245 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T14:14:46.421 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 30 14:14 /home/ubuntu/cephtest/cephadm 2024-03-30T14:14:46.421 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T14:14:46.437 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T14:14:46.461 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-30T14:14:46.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-30T14:14:46.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-30T14:14:47.348 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-30T14:14:47.375 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-30T14:15:10.008 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-30T14:15:10.008 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-30T14:15:10.008 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-30T14:15:10.008 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-03-30T14:15:10.009 INFO:teuthology.orchestra.run.smithi193.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-30T14:15:10.009 INFO:teuthology.orchestra.run.smithi193.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-30T14:15:10.009 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-30T14:15:10.009 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-03-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi031.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi031.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:15:11.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-03-30T14:15:11.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-03-30T14:15:11.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-03-30T14:15:11.144 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-03-30T14:15:11.184 INFO:tasks.cephadm:Writing seed config... 2024-03-30T14:15:11.185 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-30T14:15:11.185 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-30T14:15:11.185 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-30T14:15:11.185 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-30T14:15:11.185 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-30T14:15:11.186 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-30T14:15:11.186 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-30T14:15:11.186 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-30T14:15:11.186 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:15:11.186 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-30T14:15:11.201 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dc60522e-ee9f-11ee-b647-cb9ed24678a4 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-30T14:15:11.201 DEBUG:teuthology.orchestra.run.smithi031:mon.smithi031> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service 2024-03-30T14:15:11.244 INFO:tasks.cephadm:Bootstrapping... 2024-03-30T14:15:11.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:15:11.445 INFO:teuthology.orchestra.run.smithi031.stderr:-------------------------------------------------------------------------------- 2024-03-30T14:15:11.445 INFO:teuthology.orchestra.run.smithi031.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'dc60522e-ee9f-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2024-03-30T14:15:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: 4.9.4-dev 2024-03-30T14:15:11.470 INFO:teuthology.orchestra.run.smithi031.stderr:Verifying podman|docker is present... 2024-03-30T14:15:11.491 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: 4.9.4-dev 2024-03-30T14:15:11.493 INFO:teuthology.orchestra.run.smithi031.stderr:Verifying lvm2 is present... 2024-03-30T14:15:11.493 INFO:teuthology.orchestra.run.smithi031.stderr:Verifying time synchronization is in place... 2024-03-30T14:15:11.500 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T14:15:11.506 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: inactive 2024-03-30T14:15:11.513 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: enabled 2024-03-30T14:15:11.519 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: active 2024-03-30T14:15:11.519 INFO:teuthology.orchestra.run.smithi031.stderr:Unit chronyd.service is enabled and running 2024-03-30T14:15:11.519 INFO:teuthology.orchestra.run.smithi031.stderr:Repeating the final host check... 2024-03-30T14:15:11.539 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: 4.9.4-dev 2024-03-30T14:15:11.541 INFO:teuthology.orchestra.run.smithi031.stderr:podman (/bin/podman) version 4.9.4 is present 2024-03-30T14:15:11.541 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl is present 2024-03-30T14:15:11.542 INFO:teuthology.orchestra.run.smithi031.stderr:lvcreate is present 2024-03-30T14:15:11.547 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T14:15:11.553 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: inactive 2024-03-30T14:15:11.560 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: enabled 2024-03-30T14:15:11.565 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: active 2024-03-30T14:15:11.566 INFO:teuthology.orchestra.run.smithi031.stderr:Unit chronyd.service is enabled and running 2024-03-30T14:15:11.566 INFO:teuthology.orchestra.run.smithi031.stderr:Host looks OK 2024-03-30T14:15:11.566 INFO:teuthology.orchestra.run.smithi031.stderr:Cluster fsid: dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:11.566 INFO:teuthology.orchestra.run.smithi031.stderr:Acquiring lock 140476245171600 on /run/cephadm/dc60522e-ee9f-11ee-b647-cb9ed24678a4.lock 2024-03-30T14:15:11.567 INFO:teuthology.orchestra.run.smithi031.stderr:Lock 140476245171600 acquired on /run/cephadm/dc60522e-ee9f-11ee-b647-cb9ed24678a4.lock 2024-03-30T14:15:11.567 INFO:teuthology.orchestra.run.smithi031.stderr:Verifying IP 172.21.15.31 port 3300 ... 2024-03-30T14:15:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:Verifying IP 172.21.15.31 port 6789 ... 2024-03-30T14:15:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:Base mon IP is 172.21.15.31, final addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-03-30T14:15:11.570 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.31 metric 100 2024-03-30T14:15:11.570 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2024-03-30T14:15:11.572 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-30T14:15:11.573 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-30T14:15:11.573 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-30T14:15:11.575 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-30T14:15:11.575 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-30T14:15:11.575 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-30T14:15:11.575 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-30T14:15:11.575 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-03-30T14:15:11.575 INFO:teuthology.orchestra.run.smithi031.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-30T14:15:11.576 INFO:teuthology.orchestra.run.smithi031.stderr:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-03-30T14:15:11.576 INFO:teuthology.orchestra.run.smithi031.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-30T14:15:11.577 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-30T14:15:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-30T14:15:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Getting image source signatures 2024-03-30T14:15:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-30T14:15:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-30T14:15:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-30T14:15:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-30T14:15:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-30T14:15:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-30T14:15:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: Writing manifest to image destination 2024-03-30T14:15:12.355 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-30T14:15:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-30T14:15:13.178 INFO:teuthology.orchestra.run.smithi031.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-30T14:15:13.179 INFO:teuthology.orchestra.run.smithi031.stderr:Extracting ceph user uid/gid from container image... 2024-03-30T14:15:13.586 INFO:teuthology.orchestra.run.smithi031.stderr:stat: 167 167 2024-03-30T14:15:14.050 INFO:teuthology.orchestra.run.smithi031.stderr:Creating initial keys... 2024-03-30T14:15:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph-authtool: AQByHghm/iXRFBAAU4Io5GSR8XyRcQWh73mjmQ== 2024-03-30T14:15:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph-authtool: AQBzHghmqINHChAAIK9wlTLNdOhakbGB9+/Dnw== 2024-03-30T14:15:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph-authtool: AQBzHghmkyZaOhAA4v/nqAFc8n4Yc3yMQIsyfw== 2024-03-30T14:15:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:Creating initial monmap... 2024-03-30T14:15:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T14:15:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-30T14:15:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T14:15:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:monmaptool for smithi031 [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T14:15:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:setting min_mon_release = octopus 2024-03-30T14:15:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/monmaptool: set fsid to dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T14:15:17.280 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:15:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:Creating mon... 2024-03-30T14:15:18.086 INFO:teuthology.orchestra.run.smithi031.stderr:create mon.smithi031 on 2024-03-30T14:15:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-30T14:15:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4.target. 2024-03-30T14:15:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4.target. 2024-03-30T14:15:19.151 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Failed to reset failed state of unit ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service: Unit ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service not loaded. 2024-03-30T14:15:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4.target.wants/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service → /etc/systemd/system/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@.service. 2024-03-30T14:15:19.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:19 smithi031 podman[24545]: 2024-03-30 14:15:19.534621845 +0000 UTC m=+0.018102071 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-30T14:15:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: disabled 2024-03-30T14:15:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: inactive 2024-03-30T14:15:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:firewalld.service is not enabled 2024-03-30T14:15:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:Not possible to enable service . firewalld.service is not available 2024-03-30T14:15:20.284 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for mon to start... 2024-03-30T14:15:20.284 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for mon... 2024-03-30T14:15:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:20 smithi031 podman[24545]: 2024-03-30 14:15:20.054181676 +0000 UTC m=+0.537661931 container create 4e92d68308cd5c9de91761186431ac41c765ba4b336610f72f29c776154b91b5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., release=754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, ceph=True, io.openshift.expose-services=, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-30T14:15:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:20 smithi031 podman[24545]: 2024-03-30 14:15:20.214928338 +0000 UTC m=+0.698408564 container init 4e92d68308cd5c9de91761186431ac41c765ba4b336610f72f29c776154b91b5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, release=754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git) 2024-03-30T14:15:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:20 smithi031 podman[24545]: 2024-03-30 14:15:20.219534594 +0000 UTC m=+0.703014813 container start 4e92d68308cd5c9de91761186431ac41c765ba4b336610f72f29c776154b91b5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, ceph=True, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD) 2024-03-30T14:15:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:20 smithi031 ceph-mon[24564]: mkfs dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:20 smithi031 ceph-mon[24564]: mon.smithi031 is new leader, mons smithi031 in quorum (ranks 0) 2024-03-30T14:15:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:20 smithi031 bash[24545]: 4e92d68308cd5c9de91761186431ac41c765ba4b336610f72f29c776154b91b5 2024-03-30T14:15:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:20 smithi031 systemd[1]: Started Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:15:20.844 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: cluster: 2024-03-30T14:15:20.844 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: id: dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: services: 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi031 (age 0.594559s) 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: data: 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: pgs: 2024-03-30T14:15:20.845 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:mon is available 2024-03-30T14:15:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:Assimilating anything we can from ceph.conf... 2024-03-30T14:15:21.579 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:21 smithi031 ceph-mon[24564]: mon.smithi031 is new leader, mons smithi031 in quorum (ranks 0) 2024-03-30T14:15:21.580 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:21 smithi031 ceph-mon[24564]: monmap e1: 1 mons at {smithi031=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} 2024-03-30T14:15:21.580 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:21 smithi031 ceph-mon[24564]: fsmap 2024-03-30T14:15:21.580 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:21 smithi031 ceph-mon[24564]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T14:15:21.580 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:21 smithi031 ceph-mon[24564]: mgrmap e1: no daemons active 2024-03-30T14:15:21.580 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:21 smithi031 ceph-mon[24564]: from='client.? 172.21.15.31:0/1581093977' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-30T14:15:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: [global] 2024-03-30T14:15:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: fsid = dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-03-30T14:15:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: [mgr] 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: [osd] 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-30T14:15:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-30T14:15:22.352 INFO:teuthology.orchestra.run.smithi031.stderr:Generating new minimal ceph.conf... 2024-03-30T14:15:22.556 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:22 smithi031 ceph-mon[24564]: from='client.? 172.21.15.31:0/4220931972' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T14:15:22.557 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:22 smithi031 ceph-mon[24564]: from='client.? 172.21.15.31:0/4220931972' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-30T14:15:23.343 INFO:teuthology.orchestra.run.smithi031.stderr:Restarting the monitor... 2024-03-30T14:15:23.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 ceph-mon[24564]: from='client.? 172.21.15.31:0/1552794321' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:15:23.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 systemd[1]: Stopping Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:15:23.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 bash[24784]: Error: no container with name or ID "ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon.smithi031" found: no such container 2024-03-30T14:15:23.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[24560]: 2024-03-30T14:15:23.510+0000 7f24b5c22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi031 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:15:23.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[24560]: 2024-03-30T14:15:23.510+0000 7f24b5c22700 -1 mon.smithi031@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T14:15:23.776 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 podman[24798]: 2024-03-30 14:15:23.520988003 +0000 UTC m=+0.115954864 container died 4e92d68308cd5c9de91761186431ac41c765ba4b336610f72f29c776154b91b5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, name=centos-stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, RELEASE=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-30T14:15:24.026 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 podman[24798]: 2024-03-30 14:15:23.859674002 +0000 UTC m=+0.454640913 container cleanup 4e92d68308cd5c9de91761186431ac41c765ba4b336610f72f29c776154b91b5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, com.redhat.component=centos-stream-container, version=8, distribution-scope=public, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8) 2024-03-30T14:15:24.026 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:23 smithi031 bash[24798]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031 2024-03-30T14:15:24.282 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 podman[24813]: 2024-03-30 14:15:24.035437523 +0000 UTC m=+0.507676571 container remove 4e92d68308cd5c9de91761186431ac41c765ba4b336610f72f29c776154b91b5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.openshift.expose-services=, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.buildah.version=1.19.8) 2024-03-30T14:15:24.282 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 bash[24825]: Error: no container with name or ID "ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon.smithi031" found: no such container 2024-03-30T14:15:24.282 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service: Deactivated successfully. 2024-03-30T14:15:24.282 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 systemd[1]: Stopped Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:15:24.282 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 systemd[1]: Starting Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:15:24.282 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 podman[24893]: 2024-03-30 14:15:24.277698323 +0000 UTC m=+0.102142726 container create 1cdb8d9d80b4568668e0861d6ffe9c3f6331b08a1f2c88439a91c9e950436b64 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, version=8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.expose-services=, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-type=git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream) 2024-03-30T14:15:24.283 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 podman[24893]: 2024-03-30 14:15:24.193415428 +0000 UTC m=+0.017859845 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 podman[24893]: 2024-03-30 14:15:24.513476525 +0000 UTC m=+0.337920928 container init 1cdb8d9d80b4568668e0861d6ffe9c3f6331b08a1f2c88439a91c9e950436b64 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vendor=Red Hat, Inc., io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 podman[24893]: 2024-03-30 14:15:24.5166346 +0000 UTC m=+0.341079011 container start 1cdb8d9d80b4568668e0861d6ffe9c3f6331b08a1f2c88439a91c9e950436b64 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, com.redhat.component=centos-stream-container, version=8, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8) 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: pidfile_write: ignore empty --pid-file 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: load: jerasure load: lrc 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: RocksDB version: 6.15.5 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Compile date Apr 18 2022 2024-03-30T14:15:24.532 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: DB SUMMARY 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: DB Session ID: 6SKENPW0TZL7M9X2CE2P 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: CURRENT file: CURRENT 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi031/store.db dir, Total Num: 1, files: 000008.sst 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi031/store.db: 000010.log size: 82715 ; 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.error_if_exists: 0 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.create_if_missing: 0 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.paranoid_checks: 1 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.env: 0x55602c4c6860 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.fs: Posix File System 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.info_log: 0x55602d2b1fa0 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.statistics: (nil) 2024-03-30T14:15:24.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.use_fsync: 0 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_log_file_size: 0 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.allow_fallocate: 1 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.use_direct_reads: 0 2024-03-30T14:15:24.534 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T14:15:24.584 INFO:teuthology.orchestra.run.smithi031.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-30T14:15:24.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T14:15:24.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.db_log_dir: 2024-03-30T14:15:24.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi031/store.db 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.write_buffer_manager: 0x55602d3a2330 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T14:15:24.793 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.unordered_write: 0 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.row_cache: None 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.wal_filter: None 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T14:15:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.preserve_deletes: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.two_write_queues: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.atomic_flush: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.log_readahead_size: 0 2024-03-30T14:15:24.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T14:15:24.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T14:15:24.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T14:15:24.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T14:15:24.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T14:15:24.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T14:15:24.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_background_jobs: 2 2024-03-30T14:15:24.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_background_compactions: -1 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_subcompactions: 1 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_open_files: -1 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T14:15:24.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_background_flushes: -1 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Compression algorithms supported: 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kZSTD supported: 0 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kXpressCompression supported: 0 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kLZ4Compression supported: 1 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kBZip2Compression supported: 0 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kZlibCompression supported: 1 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: kSnappyCompression supported: 1 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi031/store.db/MANIFEST-000009 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.merge_operator: 2024-03-30T14:15:24.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_filter: None 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55602d27dd00) 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_type: 0 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: data_block_index_type: 0 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_shortening: 1 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: hash_index_allow_collision: 1 2024-03-30T14:15:24.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: checksum: 1 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: no_block_cache: 0 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache: 0x55602d2e8170 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_options: 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: capacity : 536870912 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: num_shard_bits : 4 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: strict_capacity_limit : 0 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_compressed: (nil) 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: persistent_cache: (nil) 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_size: 4096 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_size_deviation: 10 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_restart_interval: 16 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_block_restart_interval: 1 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: metadata_block_size: 4096 2024-03-30T14:15:24.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: partition_filters: 0 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: use_delta_encoding: 1 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: whole_key_filtering: 1 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: verify_compression: 0 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: format_version: 4 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: enable_index_compression: 1 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_align: 0 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression: NoCompression 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.num_levels: 7 2024-03-30T14:15:24.801 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T14:15:24.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T14:15:24.803 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.arena_block_size: 4194304 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T14:15:24.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.table_properties_collectors: 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.inplace_update_support: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.bloom_locality: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.max_successive_merges: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T14:15:24.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.ttl: 2592000 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.enable_blob_files: false 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.min_blob_size: 0 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi031/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808124542329, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808124543036, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79397, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77661, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71970, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711808124, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "df62afca-be6c-4f49-8eb2-a21bae5505ff", "db_session_id": "6SKENPW0TZL7M9X2CE2P"}} 2024-03-30T14:15:24.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808124543366, "job": 1, "event": "recovery_finished"} 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi031/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55602d2cea80 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: DB pointer 0x55602d2de000 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** DB Stats ** 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:15:24.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.000 0 0 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.000 0 0 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.000 0 0 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.2 0.00 0.00 1 0.000 0 0 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T14:15:24.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative compaction: 0.00 GB write, 24.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval compaction: 0.00 GB write, 24.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.000 0 0 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.000 0 0 2024-03-30T14:15:24.809 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.2 0.00 0.00 1 0.000 0 0 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative compaction: 0.00 GB write, 24.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:15:24.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: starting mon.smithi031 rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi031 fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???) e1 preinit fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???).mds e1 new map 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???).mds e1 print_map 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: e1 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: legacy client fscid: -1 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: No filesystems configured 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:15:24.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 bash[24893]: 1cdb8d9d80b4568668e0861d6ffe9c3f6331b08a1f2c88439a91c9e950436b64 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 systemd[1]: Started Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mon.smithi031 is new leader, mons smithi031 in quorum (ranks 0) 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: monmap e1: 1 mons at {smithi031=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: fsmap 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T14:15:24.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:24 smithi031 ceph-mon[24911]: mgrmap e1: no daemons active 2024-03-30T14:15:26.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:26 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1972618831' entity='client.admin' 2024-03-30T14:15:30.640 INFO:teuthology.orchestra.run.smithi031.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-30T14:15:30.640 INFO:teuthology.orchestra.run.smithi031.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:15:30.640 INFO:teuthology.orchestra.run.smithi031.stderr:Creating mgr... 2024-03-30T14:15:30.640 INFO:teuthology.orchestra.run.smithi031.stderr:Verifying port 9283 ... 2024-03-30T14:15:30.969 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Failed to reset failed state of unit ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mgr.smithi031.lkrchi.service: Unit ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mgr.smithi031.lkrchi.service not loaded. 2024-03-30T14:15:30.976 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4.target.wants/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mgr.smithi031.lkrchi.service → /etc/systemd/system/ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@.service. 2024-03-30T14:15:35.065 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: disabled 2024-03-30T14:15:35.072 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: inactive 2024-03-30T14:15:35.073 INFO:teuthology.orchestra.run.smithi031.stderr:firewalld.service is not enabled 2024-03-30T14:15:35.074 INFO:teuthology.orchestra.run.smithi031.stderr:Not possible to enable service . firewalld.service is not available 2024-03-30T14:15:35.082 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: disabled 2024-03-30T14:15:35.090 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: inactive 2024-03-30T14:15:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:firewalld.service is not enabled 2024-03-30T14:15:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-30T14:15:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for mgr to start... 2024-03-30T14:15:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for mgr... 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: { 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "fsid": "dc60522e-ee9f-11ee-b647-cb9ed24678a4", 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "health": { 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "checks": {}, 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mutes": [] 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-30T14:15:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum": [ 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 0 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "smithi031" 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "monmap": { 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osdmap": { 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-30T14:15:36.318 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-30T14:15:36.318 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-30T14:15:36.318 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-30T14:15:36.321 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-30T14:15:36.321 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-30T14:15:36.321 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:36.321 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "pgmap": { 2024-03-30T14:15:36.321 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-30T14:15:36.321 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-30T14:15:36.321 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "fsmap": { 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-30T14:15:36.322 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "available": false, 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "modules": [ 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "iostat", 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "nfs", 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "restful" 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "services": {} 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "servicemap": { 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "modified": "2024-03-30T14:15:20.259091+0000", 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "services": {} 2024-03-30T14:15:36.323 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:36.324 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-30T14:15:36.324 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: } 2024-03-30T14:15:36.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:36 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2929719449' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T14:15:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:mgr not available, waiting (1/15)... 2024-03-30T14:15:39.419 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:39.419 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: { 2024-03-30T14:15:39.419 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "fsid": "dc60522e-ee9f-11ee-b647-cb9ed24678a4", 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "health": { 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "checks": {}, 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mutes": [] 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum": [ 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 0 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "smithi031" 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum_age": 14, 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "monmap": { 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osdmap": { 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "pgmap": { 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-30T14:15:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "fsmap": { 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-30T14:15:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "available": false, 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "modules": [ 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "iostat", 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "nfs", 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "restful" 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "services": {} 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "servicemap": { 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "modified": "2024-03-30T14:15:20.259091+0000", 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "services": {} 2024-03-30T14:15:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-30T14:15:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: } 2024-03-30T14:15:39.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:39 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3341121033' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T14:15:40.043 INFO:teuthology.orchestra.run.smithi031.stderr:mgr not available, waiting (2/15)... 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: Activating manager daemon smithi031.lkrchi 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: mgrmap e2: smithi031.lkrchi(active, starting, since 0.00288902s) 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:40.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:40 smithi031 ceph-mon[24911]: from='mgr.14100 172.21.15.31:0/371023122' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:42.435 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:42 smithi031 ceph-mon[24911]: mgrmap e3: smithi031.lkrchi(active, since 1.00702s) 2024-03-30T14:15:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: { 2024-03-30T14:15:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "fsid": "dc60522e-ee9f-11ee-b647-cb9ed24678a4", 2024-03-30T14:15:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "health": { 2024-03-30T14:15:42.734 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "checks": {}, 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mutes": [] 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum": [ 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 0 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "smithi031" 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "quorum_age": 18, 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "monmap": { 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-30T14:15:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osdmap": { 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "pgmap": { 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-30T14:15:42.736 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "fsmap": { 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "available": true, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "modules": [ 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "iostat", 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "nfs", 2024-03-30T14:15:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "restful" 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ], 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "services": {} 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "servicemap": { 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "modified": "2024-03-30T14:15:20.259091+0000", 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "services": {} 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: }, 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-30T14:15:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: } 2024-03-30T14:15:43.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:43 smithi031 ceph-mon[24911]: mgrmap e4: smithi031.lkrchi(active, since 2s) 2024-03-30T14:15:43.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:43 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1505662203' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T14:15:43.358 INFO:teuthology.orchestra.run.smithi031.stderr:mgr is available 2024-03-30T14:15:43.963 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:43.963 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: [global] 2024-03-30T14:15:43.963 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: fsid = dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:43.963 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: [mgr] 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: [osd] 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-30T14:15:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-30T14:15:44.472 INFO:teuthology.orchestra.run.smithi031.stderr:Enabling cephadm module... 2024-03-30T14:15:44.594 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:44 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2742935772' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T14:15:45.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:45 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/108509263' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-30T14:15:46.670 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:46 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/108509263' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-30T14:15:46.670 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:46 smithi031 ceph-mon[24911]: mgrmap e5: smithi031.lkrchi(active, since 5s) 2024-03-30T14:15:46.671 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: { 2024-03-30T14:15:46.672 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-30T14:15:46.672 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "available": true, 2024-03-30T14:15:46.672 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "active_name": "smithi031.lkrchi", 2024-03-30T14:15:46.672 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-30T14:15:46.672 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: } 2024-03-30T14:15:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for the mgr to restart... 2024-03-30T14:15:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for mgr epoch 5... 2024-03-30T14:15:47.553 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:47 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1851666739' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T14:15:52.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:51 smithi031 ceph-mon[24911]: Active manager daemon smithi031.lkrchi restarted 2024-03-30T14:15:52.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:51 smithi031 ceph-mon[24911]: Activating manager daemon smithi031.lkrchi 2024-03-30T14:15:52.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:51 smithi031 ceph-mon[24911]: osdmap e2: 0 total, 0 up, 0 in 2024-03-30T14:15:52.633 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: { 2024-03-30T14:15:52.633 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-30T14:15:52.634 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "initialized": true 2024-03-30T14:15:52.634 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: } 2024-03-30T14:15:52.937 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: mgrmap e6: smithi031.lkrchi(active, starting, since 0.0522086s) 2024-03-30T14:15:52.937 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:15:52.937 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:15:52.937 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:15:52.937 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:15:52.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:52 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:15:53.239 INFO:teuthology.orchestra.run.smithi031.stderr:mgr epoch 5 is available 2024-03-30T14:15:53.240 INFO:teuthology.orchestra.run.smithi031.stderr:Setting orchestrator backend to cephadm... 2024-03-30T14:15:53.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:53 smithi031 ceph-mon[24911]: mgrmap e7: smithi031.lkrchi(active, since 1.05944s) 2024-03-30T14:15:53.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:53 smithi031 ceph-mon[24911]: [30/Mar/2024:14:15:53] ENGINE Bus STARTING 2024-03-30T14:15:53.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:53 smithi031 ceph-mon[24911]: [30/Mar/2024:14:15:53] ENGINE Serving on https://172.21.15.31:7150 2024-03-30T14:15:53.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:53 smithi031 ceph-mon[24911]: [30/Mar/2024:14:15:53] ENGINE Bus STARTED 2024-03-30T14:15:53.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:53 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:53.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:53 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:54 smithi031 ceph-mon[24911]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:15:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:54 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:54 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:54 smithi031 ceph-mon[24911]: mgrmap e8: smithi031.lkrchi(active, since 2s) 2024-03-30T14:15:55.108 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: value unchanged 2024-03-30T14:15:55.660 INFO:teuthology.orchestra.run.smithi031.stderr:Generating ssh key... 2024-03-30T14:15:55.960 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:55 smithi031 ceph-mon[24911]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:15:57.718 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJb0iSCSLnpR2uaIjqTjX4olDUbihxkELqTUMmbdrDYmJBSQTnLuVPzAiVRyMxFU1qJOqQbiKCm4L9bkrndRhSK22249VPWYWPnEUKWRwK99uh2Pl1hbx7+Jxg3axBjupTAikOFX+SjtEdO2bPrz4lg7nvU2xptlmyIduf8Ob79ho4k64mGMwXZumVM9eSucqsBgnMY0p6u9Nw3GQXsPqazIuwE4YlnXFKDGJGcjGFrdTuYDNwNGzJtGBFfYi9Hzn5QNFC9JXGwaxRp0yiApjkCC34uxxv/x3E0FJXNPQXPFXVw3/IOikWGTygrdLof3OFV8linOYdFKPonTtVHNkI5k/lUE/lP2cSQ/xHhBcQR2pjKpjph5C2DfFhjjN9i6FuKT2y+l2ODq6ppi6aGYLQvyaiP+9hDf8KZBQ1IMyzWr4QhYgQbUrKvTWtT898AUHfmfOUlkLa6pP1/HEyVMycj80x9i/6PVis1u8XrgD19VuhVgT4eJnjbw2XSH2APsk= ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:15:57.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:57 smithi031 ceph-mon[24911]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:15:57.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:57 smithi031 ceph-mon[24911]: Generating ssh key... 2024-03-30T14:15:57.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:57 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:57.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:57 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:15:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:15:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:Adding key to root@localhost authorized_keys... 2024-03-30T14:15:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:Adding host smithi031... 2024-03-30T14:15:58.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:15:58 smithi031 ceph-mon[24911]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:15:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Added host 'smithi031' with addr '172.21.15.31' 2024-03-30T14:16:00.546 INFO:teuthology.orchestra.run.smithi031.stderr:Deploying mon service with default placement... 2024-03-30T14:16:00.805 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:00 smithi031 ceph-mon[24911]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:00.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:00 smithi031 ceph-mon[24911]: Deploying cephadm binary to smithi031 2024-03-30T14:16:00.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:00 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:00.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:00 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:16:01.222 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-30T14:16:01.852 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:01 smithi031 ceph-mon[24911]: Added host smithi031 2024-03-30T14:16:01.853 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:01 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:02.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:02 smithi031 ceph-mon[24911]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:02.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:02 smithi031 ceph-mon[24911]: Saving service mon spec with placement count:5 2024-03-30T14:16:02.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:02 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:02.904 INFO:teuthology.orchestra.run.smithi031.stderr:Deploying mgr service with default placement... 2024-03-30T14:16:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-30T14:16:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:Deploying crash service with default placement... 2024-03-30T14:16:04.737 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:04 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:04.738 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:04 smithi031 ceph-mon[24911]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:04.738 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:04 smithi031 ceph-mon[24911]: Saving service mgr spec with placement count:2 2024-03-30T14:16:04.738 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:04 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Scheduled crash update... 2024-03-30T14:16:05.859 INFO:teuthology.orchestra.run.smithi031.stderr:Deploying prometheus service with default placement... 2024-03-30T14:16:06.093 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:05 smithi031 ceph-mon[24911]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:06.094 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:05 smithi031 ceph-mon[24911]: Saving service crash spec with placement * 2024-03-30T14:16:06.094 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:05 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-03-30T14:16:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:06 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:06 smithi031 ceph-mon[24911]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:06 smithi031 ceph-mon[24911]: Saving service prometheus spec with placement count:1 2024-03-30T14:16:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:06 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:06 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:Deploying grafana service with default placement... 2024-03-30T14:16:07.747 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-03-30T14:16:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:Deploying node-exporter service with default placement... 2024-03-30T14:16:09.025 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:08 smithi031 ceph-mon[24911]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:09.026 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:08 smithi031 ceph-mon[24911]: Saving service grafana spec with placement count:1 2024-03-30T14:16:09.026 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:08 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:09.254 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-03-30T14:16:10.026 INFO:teuthology.orchestra.run.smithi031.stderr:Deploying alertmanager service with default placement... 2024-03-30T14:16:10.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:10 smithi031 ceph-mon[24911]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:10.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:10 smithi031 ceph-mon[24911]: Saving service node-exporter spec with placement * 2024-03-30T14:16:10.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:10 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:10.681 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-03-30T14:16:11.988 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:11 smithi031 ceph-mon[24911]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:11.988 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:11 smithi031 ceph-mon[24911]: Saving service alertmanager spec with placement count:1 2024-03-30T14:16:11.989 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:11 smithi031 ceph-mon[24911]: from='mgr.14118 172.21.15.31:0/1703232699' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:13.249 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:13 smithi031 ceph-mon[24911]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:13.249 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:13 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/4021081703' entity='client.admin' 2024-03-30T14:16:13.863 INFO:teuthology.orchestra.run.smithi031.stderr:Enabling the dashboard module... 2024-03-30T14:16:14.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:14 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3017634527' entity='client.admin' 2024-03-30T14:16:15.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:15 smithi031 ceph-mon[24911]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:15.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:15 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3270318508' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-30T14:16:16.460 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:16 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3270318508' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-30T14:16:16.460 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:16 smithi031 ceph-mon[24911]: mgrmap e9: smithi031.lkrchi(active, since 23s) 2024-03-30T14:16:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: { 2024-03-30T14:16:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-30T14:16:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "available": true, 2024-03-30T14:16:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "active_name": "smithi031.lkrchi", 2024-03-30T14:16:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-30T14:16:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: } 2024-03-30T14:16:17.300 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for the mgr to restart... 2024-03-30T14:16:17.300 INFO:teuthology.orchestra.run.smithi031.stderr:Waiting for mgr epoch 9... 2024-03-30T14:16:17.318 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:17 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2183444268' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T14:16:21.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:21 smithi031 ceph-mon[24911]: Active manager daemon smithi031.lkrchi restarted 2024-03-30T14:16:21.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:21 smithi031 ceph-mon[24911]: Activating manager daemon smithi031.lkrchi 2024-03-30T14:16:21.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:21 smithi031 ceph-mon[24911]: osdmap e3: 0 total, 0 up, 0 in 2024-03-30T14:16:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: { 2024-03-30T14:16:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-30T14:16:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: "initialized": true 2024-03-30T14:16:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: } 2024-03-30T14:16:22.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: mgrmap e10: smithi031.lkrchi(active, starting, since 0.0549681s) 2024-03-30T14:16:22.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:16:22.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:16:22.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:22 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:16:23.043 INFO:teuthology.orchestra.run.smithi031.stderr:mgr epoch 9 is available 2024-03-30T14:16:23.043 INFO:teuthology.orchestra.run.smithi031.stderr:Generating a dashboard self-signed certificate... 2024-03-30T14:16:23.593 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:23 smithi031 ceph-mon[24911]: mgrmap e11: smithi031.lkrchi(active, since 1.06319s) 2024-03-30T14:16:23.593 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:23 smithi031 ceph-mon[24911]: [30/Mar/2024:14:16:23] ENGINE Bus STARTING 2024-03-30T14:16:23.594 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:23 smithi031 ceph-mon[24911]: [30/Mar/2024:14:16:23] ENGINE Serving on https://172.21.15.31:7150 2024-03-30T14:16:23.594 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:23 smithi031 ceph-mon[24911]: [30/Mar/2024:14:16:23] ENGINE Bus STARTED 2024-03-30T14:16:23.594 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:23 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:23.859 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-30T14:16:24.483 INFO:teuthology.orchestra.run.smithi031.stderr:Creating initial admin user... 2024-03-30T14:16:24.743 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:24 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:24.744 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:24 smithi031 ceph-mon[24911]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:24.744 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:24 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:24.744 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:24 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:24.744 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:24 smithi031 ceph-mon[24911]: mgrmap e12: smithi031.lkrchi(active, since 2s) 2024-03-30T14:16:25.463 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$VzSCysOCe3J/hnpsdxznieTX5VaK5Jd/q51C709yq3EYnKEsq7VXm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711808185, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-30T14:16:26.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:25 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:26.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:25 smithi031 ceph-mon[24911]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:26.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:25 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:26.344 INFO:teuthology.orchestra.run.smithi031.stderr:Fetching dashboard port number... 2024-03-30T14:16:27.401 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: 8443 2024-03-30T14:16:27.970 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: disabled 2024-03-30T14:16:27.977 INFO:teuthology.orchestra.run.smithi031.stderr:systemctl: inactive 2024-03-30T14:16:27.977 INFO:teuthology.orchestra.run.smithi031.stderr:firewalld.service is not enabled 2024-03-30T14:16:27.977 INFO:teuthology.orchestra.run.smithi031.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-30T14:16:27.980 INFO:teuthology.orchestra.run.smithi031.stderr:Ceph Dashboard is now available at: 2024-03-30T14:16:27.980 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:27.980 INFO:teuthology.orchestra.run.smithi031.stderr: URL: https://smithi031.front.sepia.ceph.com:8443/ 2024-03-30T14:16:27.980 INFO:teuthology.orchestra.run.smithi031.stderr: User: admin 2024-03-30T14:16:27.980 INFO:teuthology.orchestra.run.smithi031.stderr: Password: s321rb2kwi 2024-03-30T14:16:27.980 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:27.980 INFO:teuthology.orchestra.run.smithi031.stderr:Enabling autotune for osd_memory_target 2024-03-30T14:16:28.594 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:28 smithi031 ceph-mon[24911]: mgrmap e13: smithi031.lkrchi(active, since 6s) 2024-03-30T14:16:28.594 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:28 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/902874981' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-30T14:16:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr:You can access the Ceph CLI with: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: ceph telemetry on 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr:For more information see: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:30.585 INFO:teuthology.orchestra.run.smithi031.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-30T14:16:30.586 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:16:30.586 INFO:teuthology.orchestra.run.smithi031.stderr:Bootstrap complete. 2024-03-30T14:16:30.622 INFO:tasks.cephadm:Fetching config... 2024-03-30T14:16:30.623 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:16:30.623 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-30T14:16:30.639 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-30T14:16:30.639 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:16:30.639 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-30T14:16:30.695 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-30T14:16:30.695 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:16:30.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/keyring of=/dev/stdout 2024-03-30T14:16:30.763 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-30T14:16:30.763 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:16:30.763 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-30T14:16:30.819 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-30T14:16:30.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJb0iSCSLnpR2uaIjqTjX4olDUbihxkELqTUMmbdrDYmJBSQTnLuVPzAiVRyMxFU1qJOqQbiKCm4L9bkrndRhSK22249VPWYWPnEUKWRwK99uh2Pl1hbx7+Jxg3axBjupTAikOFX+SjtEdO2bPrz4lg7nvU2xptlmyIduf8Ob79ho4k64mGMwXZumVM9eSucqsBgnMY0p6u9Nw3GQXsPqazIuwE4YlnXFKDGJGcjGFrdTuYDNwNGzJtGBFfYi9Hzn5QNFC9JXGwaxRp0yiApjkCC34uxxv/x3E0FJXNPQXPFXVw3/IOikWGTygrdLof3OFV8linOYdFKPonTtVHNkI5k/lUE/lP2cSQ/xHhBcQR2pjKpjph5C2DfFhjjN9i6FuKT2y+l2ODq6ppi6aGYLQvyaiP+9hDf8KZBQ1IMyzWr4QhYgQbUrKvTWtT898AUHfmfOUlkLa6pP1/HEyVMycj80x9i/6PVis1u8XrgD19VuhVgT4eJnjbw2XSH2APsk= ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T14:16:30.897 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJb0iSCSLnpR2uaIjqTjX4olDUbihxkELqTUMmbdrDYmJBSQTnLuVPzAiVRyMxFU1qJOqQbiKCm4L9bkrndRhSK22249VPWYWPnEUKWRwK99uh2Pl1hbx7+Jxg3axBjupTAikOFX+SjtEdO2bPrz4lg7nvU2xptlmyIduf8Ob79ho4k64mGMwXZumVM9eSucqsBgnMY0p6u9Nw3GQXsPqazIuwE4YlnXFKDGJGcjGFrdTuYDNwNGzJtGBFfYi9Hzn5QNFC9JXGwaxRp0yiApjkCC34uxxv/x3E0FJXNPQXPFXVw3/IOikWGTygrdLof3OFV8linOYdFKPonTtVHNkI5k/lUE/lP2cSQ/xHhBcQR2pjKpjph5C2DfFhjjN9i6FuKT2y+l2ODq6ppi6aGYLQvyaiP+9hDf8KZBQ1IMyzWr4QhYgQbUrKvTWtT898AUHfmfOUlkLa6pP1/HEyVMycj80x9i/6PVis1u8XrgD19VuhVgT4eJnjbw2XSH2APsk= ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:16:30.929 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJb0iSCSLnpR2uaIjqTjX4olDUbihxkELqTUMmbdrDYmJBSQTnLuVPzAiVRyMxFU1qJOqQbiKCm4L9bkrndRhSK22249VPWYWPnEUKWRwK99uh2Pl1hbx7+Jxg3axBjupTAikOFX+SjtEdO2bPrz4lg7nvU2xptlmyIduf8Ob79ho4k64mGMwXZumVM9eSucqsBgnMY0p6u9Nw3GQXsPqazIuwE4YlnXFKDGJGcjGFrdTuYDNwNGzJtGBFfYi9Hzn5QNFC9JXGwaxRp0yiApjkCC34uxxv/x3E0FJXNPQXPFXVw3/IOikWGTygrdLof3OFV8linOYdFKPonTtVHNkI5k/lUE/lP2cSQ/xHhBcQR2pjKpjph5C2DfFhjjN9i6FuKT2y+l2ODq6ppi6aGYLQvyaiP+9hDf8KZBQ1IMyzWr4QhYgQbUrKvTWtT898AUHfmfOUlkLa6pP1/HEyVMycj80x9i/6PVis1u8XrgD19VuhVgT4eJnjbw2XSH2APsk= ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T14:16:30.976 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJb0iSCSLnpR2uaIjqTjX4olDUbihxkELqTUMmbdrDYmJBSQTnLuVPzAiVRyMxFU1qJOqQbiKCm4L9bkrndRhSK22249VPWYWPnEUKWRwK99uh2Pl1hbx7+Jxg3axBjupTAikOFX+SjtEdO2bPrz4lg7nvU2xptlmyIduf8Ob79ho4k64mGMwXZumVM9eSucqsBgnMY0p6u9Nw3GQXsPqazIuwE4YlnXFKDGJGcjGFrdTuYDNwNGzJtGBFfYi9Hzn5QNFC9JXGwaxRp0yiApjkCC34uxxv/x3E0FJXNPQXPFXVw3/IOikWGTygrdLof3OFV8linOYdFKPonTtVHNkI5k/lUE/lP2cSQ/xHhBcQR2pjKpjph5C2DfFhjjN9i6FuKT2y+l2ODq6ppi6aGYLQvyaiP+9hDf8KZBQ1IMyzWr4QhYgQbUrKvTWtT898AUHfmfOUlkLa6pP1/HEyVMycj80x9i/6PVis1u8XrgD19VuhVgT4eJnjbw2XSH2APsk= ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:16:30.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-30T14:16:31.306 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:31 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3419178856' entity='client.admin' 2024-03-30T14:16:33.817 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:33 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:33.817 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:33 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:16:33.817 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:33 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:33.817 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:33 smithi031 ceph-mon[24911]: Deploying daemon alertmanager.smithi031 on smithi031 2024-03-30T14:16:34.885 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-30T14:16:34.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-30T14:16:35.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:35 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1074672900' entity='client.admin' 2024-03-30T14:16:38.435 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-03-30T14:16:38.436 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T14:16:38.436 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-03-30T14:16:38.454 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T14:16:38.454 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:16:38.513 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-03-30T14:16:38.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi193 2024-03-30T14:16:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:38 smithi031 ceph-mon[24911]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:38 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:42.469 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:42 smithi031 ceph-mon[24911]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:44.486 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:44 smithi031 ceph-mon[24911]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:44.487 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:44 smithi031 ceph-mon[24911]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:45.215 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-03-30T14:16:46.030 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: Deploying cephadm binary to smithi193 2024-03-30T14:16:46.030 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:46.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:16:46.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-30T14:16:46.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:16:46.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: Deploying daemon crash.smithi031 on smithi031 2024-03-30T14:16:46.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:46.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:45 smithi031 ceph-mon[24911]: Added host smithi193 2024-03-30T14:16:46.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T14:16:47.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:47 smithi031 ceph-mon[24911]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:47.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:47 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:48.608 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:16:48.608 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-03-30T14:16:49.232 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-30T14:16:49.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-03-30T14:16:49.259 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:49 smithi031 ceph-mon[24911]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:50.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:50 smithi031 ceph-mon[24911]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:16:50.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:50 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:50.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:50 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:50.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:50 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:50.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:50 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:16:50.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:50 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:51.224 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-03-30T14:16:51.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:51 smithi031 ceph-mon[24911]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:51.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:51 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:16:51.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:51 smithi031 ceph-mon[24911]: Deploying daemon grafana.smithi031 on smithi031 2024-03-30T14:16:51.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:51 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/4192994289' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T14:16:52.134 INFO:tasks.cephadm:Adding mon.smithi031 on smithi031 2024-03-30T14:16:52.134 INFO:tasks.cephadm:Adding mon.smithi193 on smithi193 2024-03-30T14:16:52.135 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '2;smithi031:172.21.15.31=smithi031;smithi193:172.21.15.193=smithi193' 2024-03-30T14:16:52.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:52 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/4192994289' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T14:16:52.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:52 smithi031 ceph-mon[24911]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:16:52.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:52 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:53.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:53 smithi031 ceph-mon[24911]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:53.740 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-03-30T14:16:54.259 DEBUG:teuthology.orchestra.run.smithi193:mon.smithi193> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi193.service 2024-03-30T14:16:54.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:16:54.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:16:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:54 smithi031 ceph-mon[24911]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:54 smithi031 ceph-mon[24911]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi031:172.21.15.31=smithi031;smithi193:172.21.15.193=smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:16:55.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:54 smithi031 ceph-mon[24911]: Saving service mon spec with placement smithi031:172.21.15.31=smithi031;smithi193:172.21.15.193=smithi193;count:2 2024-03-30T14:16:55.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:54 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:16:56.235 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:16:56.235 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:16:56.237 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:16:57.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:56 smithi031 ceph-mon[24911]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:57.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:56 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2729000312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:16:57.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:16:57.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:16:59.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:16:58 smithi031 ceph-mon[24911]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:16:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:16:59.573 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:16:59.574 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:00.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:00 smithi031 ceph-mon[24911]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:00.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:00 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/1760335254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:01.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:01.092 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:02.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:02 smithi031 ceph-mon[24911]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:03.002 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:03.004 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:03.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:03 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:03.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:03 smithi031 ceph-mon[24911]: Deploying daemon node-exporter.smithi031 on smithi031 2024-03-30T14:17:03.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:03 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/449441648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:04.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:04.602 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:04.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:04 smithi031 ceph-mon[24911]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:06.368 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:06.368 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:06.370 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:06 smithi031 ceph-mon[24911]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:06 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:06 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T14:17:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:06 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:07.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:06 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2737191989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:07.923 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:07 smithi031 ceph-mon[24911]: from='mgr.14160 172.21.15.31:0/859599844' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T14:17:07.924 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:07 smithi031 ceph-mon[24911]: mgrmap e14: smithi031.lkrchi(active, since 45s) 2024-03-30T14:17:07.924 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:07.924 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:09.731 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:09.731 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:09.732 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:10.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:10 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/709344202' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:11.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:11.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:13.216 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:13.216 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:13.218 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:13.965 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: Active manager daemon smithi031.lkrchi restarted 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: Activating manager daemon smithi031.lkrchi 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: osdmap e5: 0 total, 0 up, 0 in 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: mgrmap e15: smithi031.lkrchi(active, starting, since 0.0526171s) 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:17:13.966 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/514455481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:13.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:14.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:14.828 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:14 smithi031 ceph-mon[24911]: [30/Mar/2024:14:17:13] ENGINE Bus STARTING 2024-03-30T14:17:14.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:14 smithi031 ceph-mon[24911]: [30/Mar/2024:14:17:13] ENGINE Serving on https://172.21.15.31:7150 2024-03-30T14:17:14.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:14 smithi031 ceph-mon[24911]: [30/Mar/2024:14:17:13] ENGINE Bus STARTED 2024-03-30T14:17:14.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:14 smithi031 ceph-mon[24911]: mgrmap e16: smithi031.lkrchi(active, since 1.05467s) 2024-03-30T14:17:14.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:16.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:16.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:15 smithi031 ceph-mon[24911]: mgrmap e17: smithi031.lkrchi(active, since 2s) 2024-03-30T14:17:17.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:17 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:17.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:17 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:17.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:17 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:17:17.940 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:17.940 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:17.942 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:18 smithi031 ceph-mon[24911]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:17:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:18 smithi031 ceph-mon[24911]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:17:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:18.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:18 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/1428507498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:19.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:19.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:20.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:20.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:20.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:17:20.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:20.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:17:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-30T14:17:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:20.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:20 smithi031 ceph-mon[24911]: Deploying daemon crash.smithi193 on smithi193 2024-03-30T14:17:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:21.406 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:21.407 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:21.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:21 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/3624949601' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:23.271 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:23.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:24 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:24 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:17:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:24 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T14:17:24.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:24 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:17:24.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:24 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:24.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:24 smithi031 ceph-mon[24911]: Deploying daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:17:26.191 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:26.191 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:26.193 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:26.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:26 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2423136131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:27 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:27 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:17:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:27 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:28.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:27 smithi031 ceph-mon[24911]: Deploying daemon mon.smithi193 on smithi193 2024-03-30T14:17:28.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:27 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:28.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:28.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:31.166 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:31.167 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:15:16.778426Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:17:31.167 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T14:17:31.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:31 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/1760995865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:33.125 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T14:17:33.125 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:17:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:17:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: mon.smithi031 calling monitor election 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1394590548' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: mon.smithi193 calling monitor election 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: mon.smithi031 is new leader, mons smithi031,smithi193 in quorum (ranks 0,1) 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: monmap e2: 2 mons at {smithi031=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],smithi193=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} 2024-03-30T14:17:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: fsmap 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: osdmap e5: 0 total, 0 up, 0 in 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: mgrmap e17: smithi031.lkrchi(active, since 23s) 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1394590548' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: overall HEALTH_OK 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1394590548' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1394590548' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:17:37.107 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:37.875 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T14:17:37.876 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:17:31.578833Z","created":"2024-03-30T14:15:16.778426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi193","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T14:17:37.877 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-03-30T14:17:38.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:37 smithi031 ceph-mon[24911]: Deploying daemon node-exporter.smithi193 on smithi193 2024-03-30T14:17:38.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:37 smithi031 ceph-mon[24911]: mgrmap e18: smithi031.lkrchi(active, since 24s), standbys: smithi193.ikzlnb 2024-03-30T14:17:38.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:37 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi193.ikzlnb", "id": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:17:38.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:37 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:17:38.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:38 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/2622045452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:39.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:38 smithi031 ceph-mon[24911]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:39.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:38 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:39.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:38 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2622045452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:17:39.141 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-30T14:17:39.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-30T14:17:40.823 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:40 smithi031 ceph-mon[24911]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:40.823 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:40 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:41.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:40 smithi193 ceph-mon[26933]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:41.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:40 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:41.113 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:17:41.113 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-03-30T14:17:41.114 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:17:41.114 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-03-30T14:17:41.780 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-30T14:17:41.780 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:17:41.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T14:17:41.810 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:17:41.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:17:41.877 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T14:17:41.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T14:17:41.907 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:41 smithi193 ceph-mon[26933]: Deploying daemon prometheus.smithi031 on smithi031 2024-03-30T14:17:41.907 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:41 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3936227830' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:41.913 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T14:17:41.913 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:17:41.984 INFO:tasks.cephadm:Deploying OSDs... 2024-03-30T14:17:41.984 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:17:41.984 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T14:17:41.999 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T14:17:41.999 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-03-30T14:17:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-30 14:17:03.254415865 +0000 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-30 14:14:22.994273859 +0000 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-30 14:14:22.994273859 +0000 2024-03-30T14:17:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-30 14:14:22.994273859 +0000 2024-03-30T14:17:42.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T14:17:42.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:41 smithi031 ceph-mon[24911]: Deploying daemon prometheus.smithi031 on smithi031 2024-03-30T14:17:42.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:41 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3936227830' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-30T14:17:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-30T14:17:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000263298 s, 1.9 MB/s 2024-03-30T14:17:42.121 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T14:17:42.177 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-30 14:17:03.254415865 +0000 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-30 14:14:23.488266281 +0000 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-30 14:14:23.488266281 +0000 2024-03-30T14:17:42.233 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-30 14:14:23.488266281 +0000 2024-03-30T14:17:42.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T14:17:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-30T14:17:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-30T14:17:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000192508 s, 2.7 MB/s 2024-03-30T14:17:42.300 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T14:17:42.358 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-30 14:17:03.254415865 +0000 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-30 14:14:23.820261188 +0000 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-30 14:14:23.820261188 +0000 2024-03-30T14:17:42.413 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-30 14:14:23.820261188 +0000 2024-03-30T14:17:42.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T14:17:42.478 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-30T14:17:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-30T14:17:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000193377 s, 2.6 MB/s 2024-03-30T14:17:42.480 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T14:17:42.535 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-03-30T14:17:42.589 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-30 14:17:03.255415849 +0000 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-30 14:14:24.181255649 +0000 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-30 14:14:24.181255649 +0000 2024-03-30T14:17:42.590 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-30 14:14:24.181255649 +0000 2024-03-30T14:17:42.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T14:17:42.657 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-30T14:17:42.657 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-30T14:17:42.657 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000189379 s, 2.7 MB/s 2024-03-30T14:17:42.659 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T14:17:42.715 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T14:17:42.715 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T14:17:42.734 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T14:17:42.734 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-03-30T14:17:42.792 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T14:17:42.792 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:42.792 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-30T14:17:42.792 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:42.792 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:42.792 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 14:17:37.199130052 +0000 2024-03-30T14:17:42.792 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 14:14:16.995822807 +0000 2024-03-30T14:17:42.793 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 14:14:16.995822807 +0000 2024-03-30T14:17:42.793 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-30 14:14:16.995822807 +0000 2024-03-30T14:17:42.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T14:17:42.863 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T14:17:42.863 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T14:17:42.863 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000254015 s, 2.0 MB/s 2024-03-30T14:17:42.864 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T14:17:42.924 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 14:17:37.200130035 +0000 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 14:14:17.328817052 +0000 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 14:14:17.328817052 +0000 2024-03-30T14:17:42.981 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-30 14:14:17.328817052 +0000 2024-03-30T14:17:42.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T14:17:43.046 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:42 smithi193 ceph-mon[26933]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:43.049 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T14:17:43.049 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T14:17:43.049 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000233149 s, 2.2 MB/s 2024-03-30T14:17:43.050 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T14:17:43.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:42 smithi031 ceph-mon[24911]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:43.108 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-03-30T14:17:43.166 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T14:17:43.166 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:43.166 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-30T14:17:43.166 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:43.166 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:43.166 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 14:17:37.200130035 +0000 2024-03-30T14:17:43.167 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 14:14:17.667811193 +0000 2024-03-30T14:17:43.167 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 14:14:17.667811193 +0000 2024-03-30T14:17:43.167 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-30 14:14:17.667811193 +0000 2024-03-30T14:17:43.167 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T14:17:43.234 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T14:17:43.234 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T14:17:43.234 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00019078 s, 2.7 MB/s 2024-03-30T14:17:43.236 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T14:17:43.291 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 14:17:37.201130018 +0000 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 14:14:18.025805006 +0000 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 14:14:18.025805006 +0000 2024-03-30T14:17:43.347 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-30 14:14:18.025805006 +0000 2024-03-30T14:17:43.348 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T14:17:43.412 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T14:17:43.413 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T14:17:43.413 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000197535 s, 2.6 MB/s 2024-03-30T14:17:43.413 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T14:17:43.469 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2024-03-30T14:17:43.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T14:17:43.786 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:43 smithi031 ceph-mon[24911]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:43.786 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:44.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:43 smithi193 ceph-mon[26933]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:44.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:46.753 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:46 smithi031 ceph-mon[24911]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:46 smithi193 ceph-mon[26933]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T14:17:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T14:17:47.037 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-30T14:17:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-30T14:17:47.037 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134033 s, 782 MB/s 2024-03-30T14:17:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-30T14:17:47.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-03-30T14:17:48.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:48 smithi031 ceph-mon[24911]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:48.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:48 smithi193 ceph-mon[26933]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:50.755 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:50 smithi031 ceph-mon[24911]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:50.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:50 smithi193 ceph-mon[26933]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:51 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:17:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:51 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:17:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:51 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:51 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:17:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:51 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:17:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:51 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:52.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:52 smithi193 ceph-mon[26933]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:52.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:52 smithi193 ceph-mon[26933]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:17:52.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:52 smithi031 ceph-mon[24911]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:52.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:52 smithi031 ceph-mon[24911]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:17:54.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:54 smithi031 ceph-mon[24911]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:55.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:54 smithi193 ceph-mon[26933]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:56.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:56 smithi031 ceph-mon[24911]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:57.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:56 smithi193 ceph-mon[26933]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:57.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:57 smithi031 ceph-mon[24911]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:57.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:57.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:17:57.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:57.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:17:58.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:57 smithi193 ceph-mon[26933]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:17:58.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:58.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:17:58.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:58.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3831537678' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "267faf1c-1984-43b9-ade7-92672d2130de"}]: dispatch 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3831537678' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "267faf1c-1984-43b9-ade7-92672d2130de"}]': finished 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/671441057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:17:58.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:58 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:17:59.062 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:59.062 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3831537678' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "267faf1c-1984-43b9-ade7-92672d2130de"}]: dispatch 2024-03-30T14:17:59.062 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3831537678' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "267faf1c-1984-43b9-ade7-92672d2130de"}]': finished 2024-03-30T14:17:59.062 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T14:17:59.062 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:17:59.063 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:17:59.063 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:17:59.063 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/671441057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:17:59.063 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:58 smithi031 ceph-mon[24911]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:18:00.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:59 smithi031 ceph-mon[24911]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:00.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:17:59 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:00.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:59 smithi193 ceph-mon[26933]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:00.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:17:59 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:02.011 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:01 smithi031 ceph-mon[24911]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:02.011 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:01 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:02.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:01 smithi193 ceph-mon[26933]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:02.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:01 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: Reconfiguring mon.smithi031 (unknown last config time)... 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:03 smithi193 ceph-mon[26933]: Reconfiguring daemon mon.smithi031 on smithi031 2024-03-30T14:18:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:18:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: Reconfiguring mon.smithi031 (unknown last config time)... 2024-03-30T14:18:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:18:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:18:03.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:03.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:03 smithi031 ceph-mon[24911]: Reconfiguring daemon mon.smithi031 on smithi031 2024-03-30T14:18:04.214 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:04 smithi031 ceph-mon[24911]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:04.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:04 smithi193 ceph-mon[26933]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:05.300 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:18:05.300 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:05.300 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:05 smithi031 ceph-mon[24911]: Deploying daemon osd.0 on smithi031 2024-03-30T14:18:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:18:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:05 smithi193 ceph-mon[26933]: Deploying daemon osd.0 on smithi031 2024-03-30T14:18:06.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:06 smithi031 ceph-mon[24911]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:06.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:06 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:06.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:06 smithi031 ceph-mon[24911]: Reconfiguring mgr.smithi031.lkrchi (unknown last config time)... 2024-03-30T14:18:06.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:06 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:18:06.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:06 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:18:06.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:06 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:06.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:06 smithi031 ceph-mon[24911]: Reconfiguring daemon mgr.smithi031.lkrchi on smithi031 2024-03-30T14:18:06.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:06 smithi193 ceph-mon[26933]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:06.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:06 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:06.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:06 smithi193 ceph-mon[26933]: Reconfiguring mgr.smithi031.lkrchi (unknown last config time)... 2024-03-30T14:18:06.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:06 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:18:06.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:06 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:18:06.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:06 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:06.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:06 smithi193 ceph-mon[26933]: Reconfiguring daemon mgr.smithi031.lkrchi on smithi031 2024-03-30T14:18:08.586 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:08 smithi031 ceph-mon[24911]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:08.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:08 smithi193 ceph-mon[26933]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:10.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:10 smithi031 ceph-mon[24911]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:10.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:10.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:10 smithi193 ceph-mon[26933]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:10.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:12.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:12 smithi031 ceph-mon[24911]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:12.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:12.838 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:12 smithi031 ceph-mon[24911]: Reconfiguring alertmanager.smithi031 (dependencies changed)... 2024-03-30T14:18:13.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:12 smithi193 ceph-mon[26933]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:13.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:13.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:12 smithi193 ceph-mon[26933]: Reconfiguring alertmanager.smithi031 (dependencies changed)... 2024-03-30T14:18:14.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:13 smithi031 ceph-mon[24911]: Reconfiguring daemon alertmanager.smithi031 on smithi031 2024-03-30T14:18:14.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:18:14.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:18:14.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:13 smithi031 ceph-mon[24911]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T14:18:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:13 smithi193 ceph-mon[26933]: Reconfiguring daemon alertmanager.smithi031 on smithi031 2024-03-30T14:18:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:18:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:18:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:13 smithi193 ceph-mon[26933]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T14:18:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:14 smithi193 ceph-mon[26933]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:14 smithi193 ceph-mon[26933]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T14:18:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:14 smithi193 ceph-mon[26933]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T14:18:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:14 smithi193 ceph-mon[26933]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:18:15.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:14 smithi031 ceph-mon[24911]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:15.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:14 smithi031 ceph-mon[24911]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T14:18:15.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:14 smithi031 ceph-mon[24911]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T14:18:15.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:15.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:14 smithi031 ceph-mon[24911]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:18:15.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:15 smithi031 ceph-mon[24911]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:18:15.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:15 smithi031 ceph-mon[24911]: osdmap e8: 1 total, 0 up, 1 in 2024-03-30T14:18:15.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:15.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:15 smithi031 ceph-mon[24911]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:15.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:15 smithi193 ceph-mon[26933]: from='osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:18:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:15 smithi193 ceph-mon[26933]: osdmap e8: 1 total, 0 up, 1 in 2024-03-30T14:18:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:15 smithi193 ceph-mon[26933]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:18:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:17.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:16 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:18:17.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:16 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:18:17.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:16 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:17.032 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:16 smithi031 ceph-mon[24911]: osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484] boot 2024-03-30T14:18:17.032 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:16 smithi031 ceph-mon[24911]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T14:18:17.032 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:16 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:16 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:18:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:16 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:18:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:16 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:16 smithi193 ceph-mon[26933]: osd.0 [v2:172.21.15.31:6802/636109484,v1:172.21.15.31:6803/636109484] boot 2024-03-30T14:18:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:16 smithi193 ceph-mon[26933]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T14:18:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:16 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:18:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:17 smithi193 ceph-mon[26933]: osdmap e10: 1 total, 1 up, 1 in 2024-03-30T14:18:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:17 smithi193 ceph-mon[26933]: pgmap v30: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:17 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:17 smithi193 ceph-mon[26933]: Reconfiguring crash.smithi031 (monmap changed)... 2024-03-30T14:18:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:17 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:18:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:17 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:17 smithi193 ceph-mon[26933]: Reconfiguring daemon crash.smithi031 on smithi031 2024-03-30T14:18:18.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:17 smithi031 ceph-mon[24911]: osdmap e10: 1 total, 1 up, 1 in 2024-03-30T14:18:18.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:17 smithi031 ceph-mon[24911]: pgmap v30: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:18.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:17 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:18.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:17 smithi031 ceph-mon[24911]: Reconfiguring crash.smithi031 (monmap changed)... 2024-03-30T14:18:18.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:17 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:18:18.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:17 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:18.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:17 smithi031 ceph-mon[24911]: Reconfiguring daemon crash.smithi031 on smithi031 2024-03-30T14:18:18.823 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2024-03-30T14:18:20.072 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:19 smithi031 ceph-mon[24911]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:20.072 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:19 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:20.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:19 smithi193 ceph-mon[26933]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:20.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:19 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:20.515 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T14:18:20.517 INFO:tasks.cephadm:Deploying osd.1 on smithi031 with /dev/vg_nvme/lv_3... 2024-03-30T14:18:20.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T14:18:21.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:21 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:21.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:21 smithi031 ceph-mon[24911]: Reconfiguring grafana.smithi031 (dependencies changed)... 2024-03-30T14:18:21.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:21 smithi031 ceph-mon[24911]: Reconfiguring daemon grafana.smithi031 on smithi031 2024-03-30T14:18:21.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:21 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:21.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:21 smithi193 ceph-mon[26933]: Reconfiguring grafana.smithi031 (dependencies changed)... 2024-03-30T14:18:21.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:21 smithi193 ceph-mon[26933]: Reconfiguring daemon grafana.smithi031 on smithi031 2024-03-30T14:18:22.176 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:22 smithi031 ceph-mon[24911]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:22 smithi193 ceph-mon[26933]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:24.303 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:24 smithi031 ceph-mon[24911]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:24.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:24 smithi193 ceph-mon[26933]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T14:18:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T14:18:25.058 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-30T14:18:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-30T14:18:25.059 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012766 s, 821 MB/s 2024-03-30T14:18:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-30T14:18:25.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-03-30T14:18:26.722 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:26 smithi031 ceph-mon[24911]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:26.723 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:26 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:26.723 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:26 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:18:26.723 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:26 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:26.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:26 smithi193 ceph-mon[26933]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:26.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:26 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:26.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:26 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:18:26.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:26 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:27.692 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:27 smithi193 ceph-mon[26933]: Reconfiguring crash.smithi193 (monmap changed)... 2024-03-30T14:18:27.693 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:27 smithi193 ceph-mon[26933]: Reconfiguring daemon crash.smithi193 on smithi193 2024-03-30T14:18:27.695 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:27 smithi031 ceph-mon[24911]: Reconfiguring crash.smithi193 (monmap changed)... 2024-03-30T14:18:27.695 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:27 smithi031 ceph-mon[24911]: Reconfiguring daemon crash.smithi193 on smithi193 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:18:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:28.832 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:28.832 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:28.832 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:18:28.832 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:18:28.832 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:28.833 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:18:28.833 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:18:28.833 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:29.796 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:29 smithi031 ceph-mon[24911]: Reconfiguring mgr.smithi193.ikzlnb (monmap changed)... 2024-03-30T14:18:29.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:29 smithi031 ceph-mon[24911]: Reconfiguring daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:18:29.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:29 smithi031 ceph-mon[24911]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:18:29.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:29 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:29.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:29 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:18:29.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:29 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:18:29.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:29 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:29.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:29 smithi193 ceph-mon[26933]: Reconfiguring mgr.smithi193.ikzlnb (monmap changed)... 2024-03-30T14:18:29.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:29 smithi193 ceph-mon[26933]: Reconfiguring daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:18:29.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:29 smithi193 ceph-mon[26933]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:18:29.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:29 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:29.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:29 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:18:29.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:29 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:18:29.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:29 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: Reconfiguring mon.smithi193 (monmap changed)... 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: Reconfiguring daemon mon.smithi193 on smithi193 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1116709745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31f794ba-8f1d-4f16-ad58-1c7064b89adb"}]: dispatch 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1116709745' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31f794ba-8f1d-4f16-ad58-1c7064b89adb"}]': finished 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T14:18:30.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:30 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: Reconfiguring mon.smithi193 (monmap changed)... 2024-03-30T14:18:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: Reconfiguring daemon mon.smithi193 on smithi193 2024-03-30T14:18:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:18:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1116709745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31f794ba-8f1d-4f16-ad58-1c7064b89adb"}]: dispatch 2024-03-30T14:18:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1116709745' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31f794ba-8f1d-4f16-ad58-1c7064b89adb"}]': finished 2024-03-30T14:18:31.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T14:18:31.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:30 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:31.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:31 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:18:31.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:31 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/505891227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:18:32.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:31 smithi193 ceph-mon[26933]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:18:32.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:31 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/505891227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:18:32.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:32 smithi031 ceph-mon[24911]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:32.979 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:32 smithi193 ceph-mon[26933]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:34.675 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:34 smithi031 ceph-mon[24911]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:34.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:34 smithi193 ceph-mon[26933]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:36.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:36 smithi031 ceph-mon[24911]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:36.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:18:36.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:37.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:36 smithi193 ceph-mon[26933]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:37.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:18:37.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:37.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:37 smithi031 ceph-mon[24911]: Deploying daemon osd.1 on smithi031 2024-03-30T14:18:38.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:37 smithi193 ceph-mon[26933]: Deploying daemon osd.1 on smithi031 2024-03-30T14:18:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:38 smithi031 ceph-mon[24911]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:39.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:38 smithi193 ceph-mon[26933]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:40.829 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:40 smithi031 ceph-mon[24911]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:41.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:40 smithi193 ceph-mon[26933]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:42.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:42 smithi031 ceph-mon[24911]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:42.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:42 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:43.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:42 smithi193 ceph-mon[26933]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:43.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:42 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:44.685 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:44 smithi031 ceph-mon[24911]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:44.720 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 1 on host 'smithi031' 2024-03-30T14:18:45.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:44 smithi193 ceph-mon[26933]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:45.396 DEBUG:teuthology.orchestra.run.smithi031:osd.1> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-30T14:18:45.398 INFO:tasks.cephadm:Deploying osd.2 on smithi031 with /dev/vg_nvme/lv_2... 2024-03-30T14:18:45.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T14:18:45.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:45.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:45.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.31:9093"}]: dispatch 2024-03-30T14:18:45.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.31:9093"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.31:3000"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.31:3000"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.31:9095"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.31:9095"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:18:45.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:45 smithi193 ceph-mon[26933]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T14:18:46.017 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:46.017 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.31:9093"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.31:9093"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.31:3000"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.31:3000"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.31:9095"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.31:9095"}]: dispatch 2024-03-30T14:18:46.018 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:46.019 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:18:46.019 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:46.019 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:18:46.019 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:45 smithi031 ceph-mon[24911]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T14:18:46.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:46 smithi193 ceph-mon[26933]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T14:18:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:46 smithi193 ceph-mon[26933]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T14:18:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:46 smithi193 ceph-mon[26933]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:18:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:46 smithi031 ceph-mon[24911]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T14:18:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:46 smithi031 ceph-mon[24911]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T14:18:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:46 smithi031 ceph-mon[24911]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:18:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:47.106 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:18:46 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[38174]: 2024-03-30T14:18:46.771+0000 7fd48c987700 -1 osd.1 0 waiting for initial osdmap 2024-03-30T14:18:47.106 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:18:46 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[38174]: 2024-03-30T14:18:46.774+0000 7fd48731e700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:18:48.001 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:47 smithi031 ceph-mon[24911]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:48.002 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:47 smithi031 ceph-mon[24911]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:18:48.002 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:47 smithi031 ceph-mon[24911]: osdmap e13: 2 total, 1 up, 2 in 2024-03-30T14:18:48.002 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:47 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:48.002 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:47 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:48.002 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:47 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:47 smithi193 ceph-mon[26933]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:47 smithi193 ceph-mon[26933]: from='osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:18:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:47 smithi193 ceph-mon[26933]: osdmap e13: 2 total, 1 up, 2 in 2024-03-30T14:18:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:47 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:47 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:47 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T14:18:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T14:18:48.731 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-30T14:18:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-30T14:18:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127319 s, 824 MB/s 2024-03-30T14:18:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-30T14:18:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:48 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:18:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:48 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:18:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:48 smithi193 ceph-mon[26933]: osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774] boot 2024-03-30T14:18:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:48 smithi193 ceph-mon[26933]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T14:18:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:48 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:18:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:48 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:18:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:48 smithi031 ceph-mon[24911]: osd.1 [v2:172.21.15.31:6810/2328121774,v1:172.21.15.31:6811/2328121774] boot 2024-03-30T14:18:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:48 smithi031 ceph-mon[24911]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T14:18:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:18:49.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-03-30T14:18:50.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:49 smithi031 ceph-mon[24911]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:50.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:49 smithi031 ceph-mon[24911]: osdmap e15: 2 total, 2 up, 2 in 2024-03-30T14:18:50.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:49 smithi193 ceph-mon[26933]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:18:50.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:49 smithi193 ceph-mon[26933]: osdmap e15: 2 total, 2 up, 2 in 2024-03-30T14:18:52.023 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:51 smithi031 ceph-mon[24911]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:18:52.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:51 smithi193 ceph-mon[26933]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:18:52.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:52 smithi031 ceph-mon[24911]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:18:52.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:52 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:18:52.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:52 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:18:52.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:52 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:53.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:52 smithi193 ceph-mon[26933]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:18:53.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:52 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:18:53.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:52 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:18:53.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:52 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:54.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:54 smithi031 ceph-mon[24911]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:18:54.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:54 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:54 smithi193 ceph-mon[26933]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:18:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:54 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:18:55.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:55 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2598549132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bd182a-ad38-4df8-a421-93fff90bbc5b"}]: dispatch 2024-03-30T14:18:55.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:55 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2598549132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60bd182a-ad38-4df8-a421-93fff90bbc5b"}]': finished 2024-03-30T14:18:55.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:55 smithi031 ceph-mon[24911]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T14:18:55.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:55 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:18:55.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:55 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2933781596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:18:56.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:55 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2598549132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bd182a-ad38-4df8-a421-93fff90bbc5b"}]: dispatch 2024-03-30T14:18:56.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:55 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2598549132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60bd182a-ad38-4df8-a421-93fff90bbc5b"}]': finished 2024-03-30T14:18:56.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:55 smithi193 ceph-mon[26933]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T14:18:56.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:55 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:18:56.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:55 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2933781596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:18:56.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:56 smithi031 ceph-mon[24911]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:18:57.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:56 smithi193 ceph-mon[26933]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:18:58.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:18:58 smithi031 ceph-mon[24911]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:18:59.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:18:58 smithi193 ceph-mon[26933]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:00.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:00.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:00.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:00.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:00.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:00.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:00.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:00.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:00.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:00.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:00.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:01.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:01 smithi193 ceph-mon[26933]: Detected new or changed devices on smithi031 2024-03-30T14:19:01.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:01 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi031 to 6434M 2024-03-30T14:19:01.993 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:01 smithi031 ceph-mon[24911]: Detected new or changed devices on smithi031 2024-03-30T14:19:01.993 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:01 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi031 to 6434M 2024-03-30T14:19:02.849 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:02 smithi031 ceph-mon[24911]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:02.849 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:02 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:02.849 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:02 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:19:02.849 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:02 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:02.979 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:02 smithi193 ceph-mon[26933]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:02.979 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:02 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:02.979 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:02 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:19:02.979 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:02 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:03.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:03 smithi193 ceph-mon[26933]: Deploying daemon osd.2 on smithi031 2024-03-30T14:19:03.856 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:03 smithi031 ceph-mon[24911]: Deploying daemon osd.2 on smithi031 2024-03-30T14:19:04.721 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:04 smithi031 ceph-mon[24911]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:05.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:04 smithi193 ceph-mon[26933]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:06.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:06 smithi031 ceph-mon[24911]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:07.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:06 smithi193 ceph-mon[26933]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:08.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:08 smithi031 ceph-mon[24911]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:08.938 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:08 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:09.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:08 smithi193 ceph-mon[26933]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:09.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:08 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:10.817 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:10 smithi031 ceph-mon[24911]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:10.817 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:11.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:10 smithi193 ceph-mon[26933]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:11.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:12.890 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:12 smithi193 ceph-mon[26933]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:12.890 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:12.890 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:12.890 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:12.890 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:12.891 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:13.061 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:12 smithi031 ceph-mon[24911]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:13.061 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:13.061 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:13.061 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:13.061 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:13.061 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:19:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:19:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:13 smithi193 ceph-mon[26933]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:19:13.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:19:13.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:19:13.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:13 smithi031 ceph-mon[24911]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:19:14.548 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 2 on host 'smithi031' 2024-03-30T14:19:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:14 smithi031 ceph-mon[24911]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:14 smithi031 ceph-mon[24911]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:19:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:14 smithi031 ceph-mon[24911]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T14:19:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:14 smithi031 ceph-mon[24911]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:19:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:14 smithi193 ceph-mon[26933]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:14 smithi193 ceph-mon[26933]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:19:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:14 smithi193 ceph-mon[26933]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T14:19:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:14 smithi193 ceph-mon[26933]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:19:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:15.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:15 smithi031 ceph-mon[24911]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:19:15.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:15 smithi031 ceph-mon[24911]: osdmap e18: 3 total, 2 up, 3 in 2024-03-30T14:19:15.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:15.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:15.894 DEBUG:teuthology.orchestra.run.smithi031:osd.2> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T14:19:15.896 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_1... 2024-03-30T14:19:15.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T14:19:16.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:15 smithi193 ceph-mon[26933]: from='osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:19:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:15 smithi193 ceph-mon[26933]: osdmap e18: 3 total, 2 up, 3 in 2024-03-30T14:19:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:16.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:16 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:19:16.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:16 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:19:16.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:16 smithi031 ceph-mon[24911]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:16.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:16 smithi031 ceph-mon[24911]: osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635] boot 2024-03-30T14:19:16.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:16 smithi031 ceph-mon[24911]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T14:19:16.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:16 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:16 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:19:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:16 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:19:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:16 smithi193 ceph-mon[26933]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:19:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:16 smithi193 ceph-mon[26933]: osd.2 [v2:172.21.15.31:6818/3163522635,v1:172.21.15.31:6819/3163522635] boot 2024-03-30T14:19:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:16 smithi193 ceph-mon[26933]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T14:19:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:16 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:19:17.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:17 smithi031 ceph-mon[24911]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T14:19:17.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:17 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-30T14:19:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:17 smithi193 ceph-mon[26933]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T14:19:18.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:17 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-30T14:19:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T14:19:18.417 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T14:19:18.417 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-30T14:19:18.417 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-30T14:19:18.417 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127415 s, 823 MB/s 2024-03-30T14:19:18.417 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-30T14:19:19.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:18 smithi193 ceph-mon[26933]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:19.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-30T14:19:19.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:18 smithi193 ceph-mon[26933]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T14:19:19.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:19:19.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:18 smithi031 ceph-mon[24911]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:19.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-30T14:19:19.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:18 smithi031 ceph-mon[24911]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T14:19:19.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:19:19.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2024-03-30T14:19:19.979 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:19 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:19:19.979 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:19 smithi031 ceph-mon[24911]: osdmap e22: 3 total, 3 up, 3 in 2024-03-30T14:19:19.979 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:19:19 smithi031 sudo[43450]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:19:19.979 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:19:19 smithi031 sudo[43450]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:19.979 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:19:19 smithi031 sudo[43450]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:20.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:19 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:19:20.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:19 smithi193 ceph-mon[26933]: osdmap e22: 3 total, 3 up, 3 in 2024-03-30T14:19:20.265 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:19:19 smithi031 sudo[43467]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:19:20.265 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:19:19 smithi031 sudo[43467]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:20.265 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:19:19 smithi031 sudo[43467]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:20.542 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43507]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:19:20.543 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43507]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:20.543 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43507]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:20.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:20 smithi031 ceph-mon[24911]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:20.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:20 smithi031 ceph-mon[24911]: osdmap e23: 3 total, 3 up, 3 in 2024-03-30T14:19:20.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:20 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:20.799 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43548]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:19:20.799 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43548]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:20.799 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43548]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:21.075 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43551]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:19:21.075 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43551]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:21.075 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:19:20 smithi031 sudo[43551]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:21.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:20 smithi193 ceph-mon[26933]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:21.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:20 smithi193 ceph-mon[26933]: osdmap e23: 3 total, 3 up, 3 in 2024-03-30T14:19:21.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:20 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:21.341 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43593]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:19:21.341 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43593]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:21.341 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43593]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:21.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43608]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:19:21.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43608]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:21.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43608]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:22.020 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43646]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:19:22.020 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43646]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:22.020 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:21 smithi031 sudo[43646]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:22.106 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:21 smithi193 sudo[29241]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:19:22.106 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:21 smithi193 sudo[29241]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:22.106 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:21 smithi193 sudo[29241]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: pgmap v75: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:19:22.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:22 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 sudo[29244]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 sudo[29244]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 sudo[29244]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: pgmap v75: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:19:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:19:22.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:22 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: mgrmap e19: smithi031.lkrchi(active, since 2m), standbys: smithi193.ikzlnb 2024-03-30T14:19:23.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: mgrmap e19: smithi031.lkrchi(active, since 2m), standbys: smithi193.ikzlnb 2024-03-30T14:19:23.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:24 smithi031 ceph-mon[24911]: pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:24.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:24 smithi193 ceph-mon[26933]: pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi031 to 4289M 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:26.245 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:26.246 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:26.246 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:26.246 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:25 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi031 to 4289M 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:26.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:25 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:27.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:26 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2505842183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9511f1ba-6389-4a0d-8779-6aed57c2f50c"}]: dispatch 2024-03-30T14:19:27.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:26 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2505842183' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9511f1ba-6389-4a0d-8779-6aed57c2f50c"}]': finished 2024-03-30T14:19:27.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:26 smithi031 ceph-mon[24911]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T14:19:27.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:26 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:27.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:26 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2505842183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9511f1ba-6389-4a0d-8779-6aed57c2f50c"}]: dispatch 2024-03-30T14:19:27.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:26 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2505842183' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9511f1ba-6389-4a0d-8779-6aed57c2f50c"}]': finished 2024-03-30T14:19:27.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:26 smithi193 ceph-mon[26933]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T14:19:27.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:26 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:28.290 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:27 smithi031 ceph-mon[24911]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:28.290 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:27 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3274601140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:19:28.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:27 smithi193 ceph-mon[26933]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:28.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:27 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3274601140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:19:30.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:30 smithi031 ceph-mon[24911]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:30.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:30 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:30.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:30 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:30 smithi193 ceph-mon[26933]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:30 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:30 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:32.831 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:32 smithi031 ceph-mon[24911]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:32.880 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:32 smithi193 ceph-mon[26933]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:33.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:33 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:19:33.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:33 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:34.020 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:33 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:19:34.020 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:33 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:34.814 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:34 smithi031 ceph-mon[24911]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:34.815 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:34 smithi031 ceph-mon[24911]: Deploying daemon osd.3 on smithi031 2024-03-30T14:19:35.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:34 smithi193 ceph-mon[26933]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:35.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:34 smithi193 ceph-mon[26933]: Deploying daemon osd.3 on smithi031 2024-03-30T14:19:36.739 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:36 smithi031 ceph-mon[24911]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:37.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:36 smithi193 ceph-mon[26933]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:38 smithi031 ceph-mon[24911]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:38 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:38 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:38 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:38.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:38 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:39.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:38 smithi193 ceph-mon[26933]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:39.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:38 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:39.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:38 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:39.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:38 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:39.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:38 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:40.924 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:40 smithi031 ceph-mon[24911]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:41.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:40 smithi193 ceph-mon[26933]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:41.222 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2024-03-30T14:19:41.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:41 smithi031 ceph-mon[24911]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:19:41.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:41.869 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-30T14:19:41.872 INFO:tasks.cephadm:Deploying osd.4 on smithi193 with /dev/vg_nvme/lv_4... 2024-03-30T14:19:41.872 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T14:19:41.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:41 smithi193 ceph-mon[26933]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:19:41.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:42.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:42 smithi193 ceph-mon[26933]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:42.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:42 smithi193 ceph-mon[26933]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T14:19:42.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:42 smithi193 ceph-mon[26933]: osdmap e25: 4 total, 3 up, 4 in 2024-03-30T14:19:42.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:42 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:42.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:42 smithi193 ceph-mon[26933]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:19:42.912 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:42 smithi031 ceph-mon[24911]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:42.913 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:42 smithi031 ceph-mon[24911]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T14:19:42.913 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:42 smithi031 ceph-mon[24911]: osdmap e25: 4 total, 3 up, 4 in 2024-03-30T14:19:42.913 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:42 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:42.913 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:42 smithi031 ceph-mon[24911]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:19:42.913 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:19:42 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[45910]: 2024-03-30T14:19:42.633+0000 7fc940b57700 -1 osd.3 0 waiting for initial osdmap 2024-03-30T14:19:42.913 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:19:42 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[45910]: 2024-03-30T14:19:42.640+0000 7fc93b4ee700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:19:43.797 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:43 smithi193 ceph-mon[26933]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:19:43.797 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:43 smithi193 ceph-mon[26933]: osdmap e26: 4 total, 3 up, 4 in 2024-03-30T14:19:43.797 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:43.797 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T14:19:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T14:19:43.798 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T14:19:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T14:19:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114105 s, 919 MB/s 2024-03-30T14:19:43.798 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 2024-03-30T14:19:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T14:19:43.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:43 smithi031 ceph-mon[24911]: from='osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-30T14:19:43.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:43 smithi031 ceph-mon[24911]: osdmap e26: 4 total, 3 up, 4 in 2024-03-30T14:19:43.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:43.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:44.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-03-30T14:19:44.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:44 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:19:44.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:44 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:19:44.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:44 smithi193 ceph-mon[26933]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:44.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:44 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:44.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:44 smithi193 ceph-mon[26933]: osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553] boot 2024-03-30T14:19:44.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:44 smithi193 ceph-mon[26933]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T14:19:44.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:44 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:44.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:44 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:19:44.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:44 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:19:44.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:44 smithi031 ceph-mon[24911]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:19:44.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:44 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:44.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:44 smithi031 ceph-mon[24911]: osd.3 [v2:172.21.15.31:6826/1435060553,v1:172.21.15.31:6827/1435060553] boot 2024-03-30T14:19:44.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:44 smithi031 ceph-mon[24911]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T14:19:44.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:44 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:19:45.776 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:45 smithi193 ceph-mon[26933]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T14:19:45.776 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:45.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:45 smithi031 ceph-mon[24911]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T14:19:45.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:46.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:46 smithi031 ceph-mon[24911]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:46.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:19:46.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:19:46.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:46.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:46 smithi193 ceph-mon[26933]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:46.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:19:46.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:19:46.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:47.897 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:47 smithi193 ceph-mon[26933]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:19:48.013 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:47 smithi031 ceph-mon[24911]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:19:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/4137813761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5844e23a-42e6-42ce-b357-ec1684cda09b"}]: dispatch 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5844e23a-42e6-42ce-b357-ec1684cda09b"}]: dispatch 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5844e23a-42e6-42ce-b357-ec1684cda09b"}]': finished 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T14:19:49.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:48 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:49.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:49.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:49.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:49.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/4137813761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5844e23a-42e6-42ce-b357-ec1684cda09b"}]: dispatch 2024-03-30T14:19:49.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5844e23a-42e6-42ce-b357-ec1684cda09b"}]: dispatch 2024-03-30T14:19:49.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5844e23a-42e6-42ce-b357-ec1684cda09b"}]': finished 2024-03-30T14:19:49.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T14:19:49.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:48 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:19:50.038 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:49 smithi031 ceph-mon[24911]: Detected new or changed devices on smithi031 2024-03-30T14:19:50.038 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:49 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi031 to 3217M 2024-03-30T14:19:50.039 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:49 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/1809371668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:19:50.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:49 smithi193 ceph-mon[26933]: Detected new or changed devices on smithi031 2024-03-30T14:19:50.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:49 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi031 to 3217M 2024-03-30T14:19:50.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:49 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/1809371668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:19:50.992 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:50 smithi031 ceph-mon[24911]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:51.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:50 smithi193 ceph-mon[26933]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:52.864 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:52 smithi193 ceph-mon[26933]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:52.864 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:52 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:52.864 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:52 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:53.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:52 smithi031 ceph-mon[24911]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:53.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:52 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:53.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:52 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:19:54.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:54 smithi193 ceph-mon[26933]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:54.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:54 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:19:54.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:54 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:54 smithi031 ceph-mon[24911]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:54 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:19:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:54 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:55.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:55 smithi193 ceph-mon[26933]: Deploying daemon osd.4 on smithi193 2024-03-30T14:19:56.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:55 smithi031 ceph-mon[24911]: Deploying daemon osd.4 on smithi193 2024-03-30T14:19:57.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:56 smithi193 ceph-mon[26933]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:57.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:56 smithi031 ceph-mon[24911]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:58.925 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:19:58 smithi193 ceph-mon[26933]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:19:59.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:19:58 smithi031 ceph-mon[24911]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:00.445 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:00 smithi193 ceph-mon[26933]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:00.445 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:00.446 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:00.446 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:00.446 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:00.446 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:00 smithi193 ceph-mon[26933]: overall HEALTH_OK 2024-03-30T14:20:00.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:00 smithi031 ceph-mon[24911]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:00.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:00.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:00.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:00.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:00.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:00 smithi031 ceph-mon[24911]: overall HEALTH_OK 2024-03-30T14:20:01.835 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 4 on host 'smithi193' 2024-03-30T14:20:02.483 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:02 smithi193 ceph-mon[26933]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:02.484 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:02 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:02.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:02 smithi031 ceph-mon[24911]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:02.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:02 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:02.799 DEBUG:teuthology.orchestra.run.smithi193:osd.4> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T14:20:02.801 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-03-30T14:20:02.802 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T14:20:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:03 smithi031 ceph-mon[24911]: from='osd.4 [v2:172.21.15.193:6800/3542342343,v1:172.21.15.193:6801/3542342343]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:20:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:03 smithi031 ceph-mon[24911]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:20:03.573 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:03 smithi193 ceph-mon[26933]: from='osd.4 [v2:172.21.15.193:6800/3542342343,v1:172.21.15.193:6801/3542342343]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:20:03.573 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:03 smithi193 ceph-mon[26933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:20:04.527 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-mon[26933]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:04.528 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-mon[26933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T14:20:04.528 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-mon[26933]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T14:20:04.528 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:04.528 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-mon[26933]: from='osd.4 [v2:172.21.15.193:6800/3542342343,v1:172.21.15.193:6801/3542342343]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:04.528 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-mon[26933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:04.528 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:04.528 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[31167]: 2024-03-30T14:20:04.206+0000 7f2b7c0f7700 -1 osd.4 0 waiting for initial osdmap 2024-03-30T14:20:04.528 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:20:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[31167]: 2024-03-30T14:20:04.211+0000 7f2b7628d700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:20:04.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:04 smithi031 ceph-mon[24911]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:04.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:04 smithi031 ceph-mon[24911]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T14:20:04.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:04 smithi031 ceph-mon[24911]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T14:20:04.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:04 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:04.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:04 smithi031 ceph-mon[24911]: from='osd.4 [v2:172.21.15.193:6800/3542342343,v1:172.21.15.193:6801/3542342343]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:04.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:04 smithi031 ceph-mon[24911]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:04.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:04 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:05.459 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:05 smithi193 ceph-mon[26933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:20:05.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:05 smithi193 ceph-mon[26933]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T14:20:05.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:05.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:05.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:05 smithi031 ceph-mon[24911]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:20:05.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:05 smithi031 ceph-mon[24911]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T14:20:05.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:05.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:05.930 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T14:20:05.930 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T14:20:05.930 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T14:20:05.930 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T14:20:05.931 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124022 s, 845 MB/s 2024-03-30T14:20:05.931 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T14:20:06.541 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:06 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:20:06.541 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:06 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:20:06.541 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:06 smithi193 ceph-mon[26933]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:06.541 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:06 smithi193 ceph-mon[26933]: osd.4 [v2:172.21.15.193:6800/3542342343,v1:172.21.15.193:6801/3542342343] boot 2024-03-30T14:20:06.541 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:06 smithi193 ceph-mon[26933]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T14:20:06.541 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:06 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:06.541 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:06 smithi193 ceph-mon[26933]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T14:20:06.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:06 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:20:06.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:06 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:20:06.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:06 smithi031 ceph-mon[24911]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-30T14:20:06.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:06 smithi031 ceph-mon[24911]: osd.4 [v2:172.21.15.193:6800/3542342343,v1:172.21.15.193:6801/3542342343] boot 2024-03-30T14:20:06.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:06 smithi031 ceph-mon[24911]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T14:20:06.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:06 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:20:06.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:06 smithi031 ceph-mon[24911]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T14:20:07.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-03-30T14:20:08.534 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:08 smithi193 ceph-mon[26933]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:08.535 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:08 smithi193 ceph-mon[26933]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T14:20:08.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:08 smithi031 ceph-mon[24911]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:08.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:08 smithi031 ceph-mon[24911]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T14:20:09.542 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T14:20:09.542 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:09.543 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:09.543 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:09.543 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:09.543 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:09.543 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:09.543 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:09.543 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:09 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:09.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:09 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:10.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:10.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: Detected new or changed devices on smithi193 2024-03-30T14:20:10.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi193 to 15936M 2024-03-30T14:20:10.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T14:20:10.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:20:10.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:20:10.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:20:10.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:10.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:10.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: Detected new or changed devices on smithi193 2024-03-30T14:20:10.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi193 to 15936M 2024-03-30T14:20:10.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T14:20:10.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:20:10.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:20:10.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:20:10.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:12.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:12 smithi193 ceph-mon[26933]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-30T14:20:12.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:12 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/2080678216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd"}]: dispatch 2024-03-30T14:20:12.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:12 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd"}]: dispatch 2024-03-30T14:20:12.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:12 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd"}]': finished 2024-03-30T14:20:12.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:12 smithi193 ceph-mon[26933]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T14:20:12.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:12 smithi031 ceph-mon[24911]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-30T14:20:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:12 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2080678216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd"}]: dispatch 2024-03-30T14:20:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:12 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd"}]: dispatch 2024-03-30T14:20:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:12 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd"}]': finished 2024-03-30T14:20:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:12 smithi031 ceph-mon[24911]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T14:20:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:13.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:13 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/4041906172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:20:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:20:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:20:13.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:13 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/4041906172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:20:13.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:20:13.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:20:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:14 smithi193 ceph-mon[26933]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-30T14:20:14.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:14 smithi031 ceph-mon[24911]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-30T14:20:16.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:16 smithi031 ceph-mon[24911]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-30T14:20:16.874 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:16 smithi193 ceph-mon[26933]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-30T14:20:18.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:18 smithi193 ceph-mon[26933]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-30T14:20:18.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:20:18.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:18.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:18 smithi031 ceph-mon[24911]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-30T14:20:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:20:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:19.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:19 smithi193 ceph-mon[26933]: Deploying daemon osd.5 on smithi193 2024-03-30T14:20:20.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:19 smithi031 ceph-mon[24911]: Deploying daemon osd.5 on smithi193 2024-03-30T14:20:20.987 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:20 smithi193 ceph-mon[26933]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:21.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:20 smithi031 ceph-mon[24911]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:22.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:22 smithi193 ceph-mon[26933]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:23.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:22 smithi031 ceph-mon[24911]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:24.149 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:23 smithi193 ceph-mon[26933]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:24.149 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:24.149 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:24.149 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:24.149 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:23 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:24.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:23 smithi031 ceph-mon[24911]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:24.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:23 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:25.847 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-03-30T14:20:26.451 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-30T14:20:26.453 INFO:tasks.cephadm:Deploying osd.6 on smithi193 with /dev/vg_nvme/lv_2... 2024-03-30T14:20:26.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T14:20:26.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:26 smithi193 ceph-mon[26933]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:26.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:26 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:26.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:26 smithi193 ceph-mon[26933]: from='osd.5 [v2:172.21.15.193:6808/71481222,v1:172.21.15.193:6809/71481222]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:20:26.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:26 smithi193 ceph-mon[26933]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:20:26.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:26 smithi031 ceph-mon[24911]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:26.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:26 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:26.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:26 smithi031 ceph-mon[24911]: from='osd.5 [v2:172.21.15.193:6808/71481222,v1:172.21.15.193:6809/71481222]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:20:26.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:26 smithi031 ceph-mon[24911]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:20:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:27 smithi031 ceph-mon[24911]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:27 smithi031 ceph-mon[24911]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T14:20:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:27 smithi031 ceph-mon[24911]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T14:20:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:27 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:27 smithi031 ceph-mon[24911]: from='osd.5 [v2:172.21.15.193:6808/71481222,v1:172.21.15.193:6809/71481222]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:27 smithi031 ceph-mon[24911]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:28.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:27 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:28.247 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[34139]: 2024-03-30T14:20:27.863+0000 7f2adc528700 -1 osd.5 0 waiting for initial osdmap 2024-03-30T14:20:28.247 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[34139]: 2024-03-30T14:20:27.868+0000 7f2ad66be700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:20:28.248 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-mon[26933]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:28.248 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-mon[26933]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T14:20:28.248 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-mon[26933]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T14:20:28.248 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:28.248 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-mon[26933]: from='osd.5 [v2:172.21.15.193:6808/71481222,v1:172.21.15.193:6809/71481222]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:28.248 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-mon[26933]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:28.248 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:27 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:29.011 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T14:20:29.012 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T14:20:29.012 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T14:20:29.012 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T14:20:29.012 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123324 s, 850 MB/s 2024-03-30T14:20:29.012 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T14:20:29.162 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:28 smithi193 ceph-mon[26933]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:20:29.163 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:28 smithi193 ceph-mon[26933]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T14:20:29.163 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:29.163 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:28 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:29.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:28 smithi031 ceph-mon[24911]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:20:29.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:28 smithi031 ceph-mon[24911]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T14:20:29.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:29.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:28 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:29.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-03-30T14:20:30.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:20:30.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:20:30.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:30.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:30.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: osd.5 [v2:172.21.15.193:6808/71481222,v1:172.21.15.193:6809/71481222] boot 2024-03-30T14:20:30.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T14:20:30.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:30.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:29 smithi193 ceph-mon[26933]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T14:20:30.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:20:30.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:20:30.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:20:30.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:30.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: osd.5 [v2:172.21.15.193:6808/71481222,v1:172.21.15.193:6809/71481222] boot 2024-03-30T14:20:30.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T14:20:30.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:20:30.356 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:29 smithi031 ceph-mon[24911]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T14:20:31.712 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:31 smithi193 ceph-mon[26933]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T14:20:32.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:31 smithi031 ceph-mon[24911]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:32.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:32 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:33.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:33.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:33.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:32 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:33.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:33 smithi193 ceph-mon[26933]: Detected new or changed devices on smithi193 2024-03-30T14:20:33.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:33 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi193 to 7968M 2024-03-30T14:20:33.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:33 smithi193 ceph-mon[26933]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T14:20:33.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:33 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:20:33.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:33 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:20:33.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:33 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:34.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:33 smithi031 ceph-mon[24911]: Detected new or changed devices on smithi193 2024-03-30T14:20:34.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:33 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi193 to 7968M 2024-03-30T14:20:34.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:33 smithi031 ceph-mon[24911]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T14:20:34.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:33 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:20:34.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:33 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:20:34.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:33 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:34.779 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:34 smithi193 ceph-mon[26933]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:34.779 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:34 smithi193 ceph-mon[26933]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:20:35.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:34 smithi031 ceph-mon[24911]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:35.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:34 smithi031 ceph-mon[24911]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:20:36.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:35 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/2423598489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6eb52d6-f435-49aa-a632-9b17aba3d5b1"}]: dispatch 2024-03-30T14:20:36.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:35 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6eb52d6-f435-49aa-a632-9b17aba3d5b1"}]: dispatch 2024-03-30T14:20:36.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:35 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6eb52d6-f435-49aa-a632-9b17aba3d5b1"}]': finished 2024-03-30T14:20:36.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:35 smithi193 ceph-mon[26933]: osdmap e45: 7 total, 6 up, 7 in 2024-03-30T14:20:36.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:35 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:35 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2423598489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6eb52d6-f435-49aa-a632-9b17aba3d5b1"}]: dispatch 2024-03-30T14:20:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:35 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6eb52d6-f435-49aa-a632-9b17aba3d5b1"}]: dispatch 2024-03-30T14:20:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:35 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6eb52d6-f435-49aa-a632-9b17aba3d5b1"}]': finished 2024-03-30T14:20:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:35 smithi031 ceph-mon[24911]: osdmap e45: 7 total, 6 up, 7 in 2024-03-30T14:20:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:35 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:37.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:36 smithi193 ceph-mon[26933]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:37.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:36 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/2001249371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:20:37.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:36 smithi031 ceph-mon[24911]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:36 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2001249371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:20:38.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:38 smithi193 ceph-mon[26933]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:39.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:38 smithi031 ceph-mon[24911]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:40.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:40 smithi193 ceph-mon[26933]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:40.907 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:40 smithi031 ceph-mon[24911]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:41.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:20:41.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:42.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:20:42.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:42.888 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:42 smithi193 ceph-mon[26933]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:42.889 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:42 smithi193 ceph-mon[26933]: Deploying daemon osd.6 on smithi193 2024-03-30T14:20:43.099 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:42 smithi031 ceph-mon[24911]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:43.099 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:42 smithi031 ceph-mon[24911]: Deploying daemon osd.6 on smithi193 2024-03-30T14:20:44.758 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:44 smithi193 ceph-mon[26933]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:45.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:44 smithi031 ceph-mon[24911]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:46.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:46 smithi193 ceph-mon[26933]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:46.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:46.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:46 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:47.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:46 smithi031 ceph-mon[24911]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:47.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:46 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:48.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:48 smithi193 ceph-mon[26933]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:49.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:48 smithi031 ceph-mon[24911]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:50.537 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 6 on host 'smithi193' 2024-03-30T14:20:50.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:50 smithi193 ceph-mon[26933]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:50.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:50 smithi193 ceph-mon[26933]: from='osd.6 [v2:172.21.15.193:6816/3087372008,v1:172.21.15.193:6817/3087372008]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:20:50.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:50 smithi193 ceph-mon[26933]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:20:50.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:50 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:50.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:50 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:50.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:50 smithi031 ceph-mon[24911]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:50.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:50 smithi031 ceph-mon[24911]: from='osd.6 [v2:172.21.15.193:6816/3087372008,v1:172.21.15.193:6817/3087372008]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:20:50.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:50 smithi031 ceph-mon[24911]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:20:50.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:50 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:50.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:50 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:51.172 DEBUG:teuthology.orchestra.run.smithi193:osd.6> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.6.service 2024-03-30T14:20:51.174 INFO:tasks.cephadm:Deploying osd.7 on smithi193 with /dev/vg_nvme/lv_1... 2024-03-30T14:20:51.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T14:20:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:51 smithi193 ceph-mon[26933]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T14:20:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:51 smithi193 ceph-mon[26933]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T14:20:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:51 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:51 smithi193 ceph-mon[26933]: from='osd.6 [v2:172.21.15.193:6816/3087372008,v1:172.21.15.193:6817/3087372008]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:51.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:51 smithi193 ceph-mon[26933]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:51.828 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:20:51 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[37122]: 2024-03-30T14:20:51.625+0000 7fc72cbf5700 -1 osd.6 0 waiting for initial osdmap 2024-03-30T14:20:51.828 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:20:51 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[37122]: 2024-03-30T14:20:51.631+0000 7fc729590700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:20:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:51 smithi031 ceph-mon[24911]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T14:20:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:51 smithi031 ceph-mon[24911]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T14:20:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:51 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:51 smithi031 ceph-mon[24911]: from='osd.6 [v2:172.21.15.193:6816/3087372008,v1:172.21.15.193:6817/3087372008]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:51.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:51 smithi031 ceph-mon[24911]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:20:52.933 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:52 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:20:52.933 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:52 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:20:52.933 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:52 smithi193 ceph-mon[26933]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:52.934 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:52 smithi193 ceph-mon[26933]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:20:52.934 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:52 smithi193 ceph-mon[26933]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T14:20:52.934 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:52 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:52.934 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:52 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:53.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:52 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:20:53.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:52 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:20:53.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:52 smithi031 ceph-mon[24911]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:53.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:52 smithi031 ceph-mon[24911]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:20:53.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:52 smithi031 ceph-mon[24911]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T14:20:53.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:52 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:53.101 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:52 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:53.311 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T14:20:53.311 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T14:20:53.311 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T14:20:53.311 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T14:20:53.311 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123192 s, 851 MB/s 2024-03-30T14:20:53.311 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T14:20:53.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:53 smithi193 ceph-mon[26933]: osd.6 [v2:172.21.15.193:6816/3087372008,v1:172.21.15.193:6817/3087372008] boot 2024-03-30T14:20:53.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:53 smithi193 ceph-mon[26933]: osdmap e48: 7 total, 7 up, 7 in 2024-03-30T14:20:53.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:53 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:54.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-03-30T14:20:54.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:53 smithi031 ceph-mon[24911]: osd.6 [v2:172.21.15.193:6816/3087372008,v1:172.21.15.193:6817/3087372008] boot 2024-03-30T14:20:54.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:53 smithi031 ceph-mon[24911]: osdmap e48: 7 total, 7 up, 7 in 2024-03-30T14:20:54.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:53 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:20:54.758 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:54 smithi193 ceph-mon[26933]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:54.758 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:54 smithi193 ceph-mon[26933]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T14:20:55.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:54 smithi031 ceph-mon[24911]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:20:55.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:54 smithi031 ceph-mon[24911]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T14:20:56.033 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:55 smithi193 ceph-mon[26933]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T14:20:56.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:55 smithi031 ceph-mon[24911]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T14:20:57.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:56 smithi193 ceph-mon[26933]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:20:57.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:56 smithi193 ceph-mon[26933]: osdmap e51: 7 total, 7 up, 7 in 2024-03-30T14:20:57.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:56 smithi031 ceph-mon[24911]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:20:57.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:56 smithi031 ceph-mon[24911]: osdmap e51: 7 total, 7 up, 7 in 2024-03-30T14:20:58.071 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T14:20:58.071 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:20:58.071 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:20:58.071 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:58.071 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:58.071 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:58.072 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:58.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:58.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:58.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:20:59.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:58 smithi193 ceph-mon[26933]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:20:59.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:58 smithi193 ceph-mon[26933]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:20:59.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:58 smithi193 ceph-mon[26933]: Detected new or changed devices on smithi193 2024-03-30T14:20:59.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:58 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi193 to 5312M 2024-03-30T14:20:59.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:58 smithi031 ceph-mon[24911]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:20:59.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:58 smithi031 ceph-mon[24911]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:20:59.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:58 smithi031 ceph-mon[24911]: Detected new or changed devices on smithi193 2024-03-30T14:20:59.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:58 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi193 to 5312M 2024-03-30T14:21:00.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:59 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/2993697893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93ea6d87-04b5-4c11-92ff-0766b9788fc0"}]: dispatch 2024-03-30T14:21:00.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:59 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93ea6d87-04b5-4c11-92ff-0766b9788fc0"}]: dispatch 2024-03-30T14:21:00.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:59 smithi193 ceph-mon[26933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93ea6d87-04b5-4c11-92ff-0766b9788fc0"}]': finished 2024-03-30T14:21:00.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:59 smithi193 ceph-mon[26933]: osdmap e53: 8 total, 7 up, 8 in 2024-03-30T14:21:00.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:59 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:00.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:20:59 smithi193 ceph-mon[26933]: from='client.? 172.21.15.193:0/2964224869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:21:00.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:59 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2993697893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93ea6d87-04b5-4c11-92ff-0766b9788fc0"}]: dispatch 2024-03-30T14:21:00.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:59 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93ea6d87-04b5-4c11-92ff-0766b9788fc0"}]: dispatch 2024-03-30T14:21:00.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:59 smithi031 ceph-mon[24911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93ea6d87-04b5-4c11-92ff-0766b9788fc0"}]': finished 2024-03-30T14:21:00.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:59 smithi031 ceph-mon[24911]: osdmap e53: 8 total, 7 up, 8 in 2024-03-30T14:21:00.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:59 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:00.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:20:59 smithi031 ceph-mon[24911]: from='client.? 172.21.15.193:0/2964224869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:21:01.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:00 smithi193 ceph-mon[26933]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-30T14:21:01.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:00 smithi031 ceph-mon[24911]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-30T14:21:02.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:02 smithi193 ceph-mon[26933]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-30T14:21:03.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:02 smithi031 ceph-mon[24911]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-30T14:21:04.908 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:04 smithi193 ceph-mon[26933]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:21:05.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:04 smithi031 ceph-mon[24911]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:21:05.928 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:21:05.929 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:06.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:21:06.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:06.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:06 smithi193 ceph-mon[26933]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T14:21:06.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:06 smithi193 ceph-mon[26933]: Deploying daemon osd.7 on smithi193 2024-03-30T14:21:07.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:06 smithi031 ceph-mon[24911]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T14:21:07.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:06 smithi031 ceph-mon[24911]: Deploying daemon osd.7 on smithi193 2024-03-30T14:21:08.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:08 smithi193 ceph-mon[26933]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-30T14:21:09.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:08 smithi031 ceph-mon[24911]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-30T14:21:10.794 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:10 smithi193 ceph-mon[26933]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:10.794 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:10.794 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:21:10.794 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:10.794 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:21:10.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:10 smithi031 ceph-mon[24911]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:10.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:10.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:21:10.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:10.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:21:12.965 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:12 smithi193 ceph-mon[26933]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:13.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:12 smithi031 ceph-mon[24911]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:13.221 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 7 on host 'smithi193' 2024-03-30T14:21:13.819 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:21:13.819 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:21:13.819 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:13.819 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:13 smithi193 ceph-mon[26933]: from='osd.7 [v2:172.21.15.193:6824/4193965660,v1:172.21.15.193:6825/4193965660]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:21:13.819 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:13 smithi193 ceph-mon[26933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:21:13.840 DEBUG:teuthology.orchestra.run.smithi193:osd.7> sudo journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.7.service 2024-03-30T14:21:13.843 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-30T14:21:13.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T14:21:14.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:21:14.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:21:14.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:14.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:13 smithi031 ceph-mon[24911]: from='osd.7 [v2:172.21.15.193:6824/4193965660,v1:172.21.15.193:6825/4193965660]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:21:14.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:13 smithi031 ceph-mon[24911]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:21:14.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:14 smithi031 ceph-mon[24911]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:14 smithi031 ceph-mon[24911]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T14:21:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:14 smithi031 ceph-mon[24911]: osdmap e54: 8 total, 7 up, 8 in 2024-03-30T14:21:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:14 smithi031 ceph-mon[24911]: from='osd.7 [v2:172.21.15.193:6824/4193965660,v1:172.21.15.193:6825/4193965660]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:21:14.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:14 smithi031 ceph-mon[24911]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:21:15.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:14 smithi193 ceph-mon[26933]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:15.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:14 smithi193 ceph-mon[26933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T14:21:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:14 smithi193 ceph-mon[26933]: osdmap e54: 8 total, 7 up, 8 in 2024-03-30T14:21:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:14 smithi193 ceph-mon[26933]: from='osd.7 [v2:172.21.15.193:6824/4193965660,v1:172.21.15.193:6825/4193965660]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:21:15.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:14 smithi193 ceph-mon[26933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:21:15.450 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[40205]: 2024-03-30T14:21:15.235+0000 7f5b15147700 -1 osd.7 0 waiting for initial osdmap 2024-03-30T14:21:15.450 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[40205]: 2024-03-30T14:21:15.243+0000 7f5b0fade700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:21:15.838 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:15 smithi031 ceph-mon[24911]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:15.838 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:15.838 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:15 smithi031 ceph-mon[24911]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:21:15.838 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:15 smithi031 ceph-mon[24911]: osdmap e55: 8 total, 7 up, 8 in 2024-03-30T14:21:15.839 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:15.839 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:15.890 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:21:16.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-mon[26933]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:21:16.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-mon[26933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T14:21:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-mon[26933]: osdmap e55: 8 total, 7 up, 8 in 2024-03-30T14:21:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:16.637 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"num_osds":8,"num_up_osds":7,"osd_up_since":1711808452,"num_in_osds":8,"osd_in_since":1711808459,"num_remapped_pgs":0} 2024-03-30T14:21:16.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:16 smithi031 ceph-mon[24911]: purged_snaps scrub starts 2024-03-30T14:21:16.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:16 smithi031 ceph-mon[24911]: purged_snaps scrub ok 2024-03-30T14:21:16.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:16 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2905467361' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T14:21:16.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:16 smithi031 ceph-mon[24911]: osd.7 [v2:172.21.15.193:6824/4193965660,v1:172.21.15.193:6825/4193965660] boot 2024-03-30T14:21:16.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:16 smithi031 ceph-mon[24911]: osdmap e56: 8 total, 8 up, 8 in 2024-03-30T14:21:16.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:16 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:16 smithi193 ceph-mon[26933]: purged_snaps scrub starts 2024-03-30T14:21:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:16 smithi193 ceph-mon[26933]: purged_snaps scrub ok 2024-03-30T14:21:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:16 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2905467361' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T14:21:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:16 smithi193 ceph-mon[26933]: osd.7 [v2:172.21.15.193:6824/4193965660,v1:172.21.15.193:6825/4193965660] boot 2024-03-30T14:21:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:16 smithi193 ceph-mon[26933]: osdmap e56: 8 total, 8 up, 8 in 2024-03-30T14:21:17.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:16 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:21:17.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T14:21:18.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:18.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: osdmap e57: 8 total, 8 up, 8 in 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:21:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:18.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: osdmap e57: 8 total, 8 up, 8 in 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:21:18.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:21:19.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:19 smithi193 ceph-mon[26933]: Detected new or changed devices on smithi193 2024-03-30T14:21:19.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:19 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi193 to 3984M 2024-03-30T14:21:19.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:19 smithi193 ceph-mon[26933]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T14:21:19.597 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:21:19.597 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:19 smithi031 ceph-mon[24911]: Detected new or changed devices on smithi193 2024-03-30T14:21:19.597 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:19 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi193 to 3984M 2024-03-30T14:21:19.597 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:19 smithi031 ceph-mon[24911]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T14:21:20.230 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1711808476,"num_in_osds":8,"osd_in_since":1711808459,"num_remapped_pgs":0} 2024-03-30T14:21:20.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:21:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:20.576 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:20 smithi031 ceph-mon[24911]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:20.576 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:20 smithi031 ceph-mon[24911]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T14:21:20.576 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:20 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2145969696' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T14:21:20.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:20 smithi193 ceph-mon[26933]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:20.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:20 smithi193 ceph-mon[26933]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T14:21:20.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:20 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2145969696' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T14:21:21.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:21 smithi031 ceph-mon[24911]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T14:21:21.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:21 smithi193 ceph-mon[26933]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T14:21:22.228 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:21:22.229 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:15:20.258619+0000","modified":"2024-03-30T14:21:20.252536+0000","last_up_change":"2024-03-30T14:21:16.231260+0000","last_in_change":"2024-03-30T14:20:59.009418+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:19:17.016586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"267faf1c-1984-43b9-ade7-92672d2130de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6803","nonce":636109484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6805","nonce":636109484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6809","nonce":636109484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6807","nonce":636109484}]},"public_addr":"172.21.15.31:6803/636109484","cluster_addr":"172.21.15.31:6805/636109484","heartbeat_back_addr":"172.21.15.31:6809/636109484","heartbeat_front_addr":"172.21.15.31:6807/636109484","state":["exists","up"]},{"osd":1,"uuid":"31f794ba-8f1d-4f16-ad58-1c7064b89adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6811","nonce":2328121774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6813","nonce":2328121774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6817","nonce":2328121774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6815","nonce":2328121774}]},"public_addr":"172.21.15.31:6811/2328121774","cluster_addr":"172.21.15.31:6813/2328121774","heartbeat_back_addr":"172.21.15.31:6817/2328121774","heartbeat_front_addr":"172.21.15.31:6815/2328121774","state":["exists","up"]},{"osd":2,"uuid":"60bd182a-ad38-4df8-a421-93fff90bbc5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6819","nonce":3163522635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6821","nonce":3163522635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6825","nonce":3163522635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6823","nonce":3163522635}]},"public_addr":"172.21.15.31:6819/3163522635","cluster_addr":"172.21.15.31:6821/3163522635","heartbeat_back_addr":"172.21.15.31:6825/3163522635","heartbeat_front_addr":"172.21.15.31:6823/3163522635","state":["exists","up"]},{"osd":3,"uuid":"9511f1ba-6389-4a0d-8779-6aed57c2f50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6827","nonce":1435060553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6829","nonce":1435060553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6833","nonce":1435060553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6831","nonce":1435060553}]},"public_addr":"172.21.15.31:6827/1435060553","cluster_addr":"172.21.15.31:6829/1435060553","heartbeat_back_addr":"172.21.15.31:6833/1435060553","heartbeat_front_addr":"172.21.15.31:6831/1435060553","state":["exists","up"]},{"osd":4,"uuid":"5844e23a-42e6-42ce-b357-ec1684cda09b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6801","nonce":3542342343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6803","nonce":3542342343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6807","nonce":3542342343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6805","nonce":3542342343}]},"public_addr":"172.21.15.193:6801/3542342343","cluster_addr":"172.21.15.193:6803/3542342343","heartbeat_back_addr":"172.21.15.193:6807/3542342343","heartbeat_front_addr":"172.21.15.193:6805/3542342343","state":["exists","up"]},{"osd":5,"uuid":"ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6809","nonce":71481222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6811","nonce":71481222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6815","nonce":71481222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6813","nonce":71481222}]},"public_addr":"172.21.15.193:6809/71481222","cluster_addr":"172.21.15.193:6811/71481222","heartbeat_back_addr":"172.21.15.193:6815/71481222","heartbeat_front_addr":"172.21.15.193:6813/71481222","state":["exists","up"]},{"osd":6,"uuid":"d6eb52d6-f435-49aa-a632-9b17aba3d5b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6817","nonce":3087372008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6819","nonce":3087372008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6823","nonce":3087372008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6821","nonce":3087372008}]},"public_addr":"172.21.15.193:6817/3087372008","cluster_addr":"172.21.15.193:6819/3087372008","heartbeat_back_addr":"172.21.15.193:6823/3087372008","heartbeat_front_addr":"172.21.15.193:6821/3087372008","state":["exists","up"]},{"osd":7,"uuid":"93ea6d87-04b5-4c11-92ff-0766b9788fc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6825","nonce":4193965660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6827","nonce":4193965660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6831","nonce":4193965660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6829","nonce":4193965660}]},"public_addr":"172.21.15.193:6825/4193965660","cluster_addr":"172.21.15.193:6827/4193965660","heartbeat_back_addr":"172.21.15.193:6831/4193965660","heartbeat_front_addr":"172.21.15.193:6829/4193965660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:18:14.485327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:18:45.834933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:19:14.406169+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:19:42.070138+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:03.342656+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:27.026452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:50.576227+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:21:14.289453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/4029290981":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6801/4196482206":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/4271842515":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6800/4196482206":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/2731736691":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/1332734773":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/3656726806":"2024-03-31T14:17:12.649056+0000","172.21.15.31:0/3329762819":"2024-03-31T14:15:51.576676+0000","172.21.15.31:6801/4035715893":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6801/2607887203":"2024-03-31T14:15:51.576676+0000","172.21.15.31:0/3988990777":"2024-03-31T14:16:21.262711+0000","172.21.15.31:6800/2607887203":"2024-03-31T14:15:51.576676+0000","172.21.15.31:0/3897097667":"2024-03-31T14:15:51.576676+0000","172.21.15.31:6800/4035715893":"2024-03-31T14:17:12.649056+0000","172.21.15.31:0/492591059":"2024-03-31T14:15:51.576676+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:21:22.551 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:22 smithi031 ceph-mon[24911]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:22.551 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:22 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2682852237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:21:22.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:22 smithi193 ceph-mon[26933]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:22.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:22 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2682852237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:21:22.976 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:19:17.016586+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-30T14:21:22.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-30T14:21:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:24.562 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:24 smithi031 ceph-mon[24911]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-30T14:21:24.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:24 smithi193 ceph-mon[26933]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-30T14:21:25.108 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-30T14:21:25.823 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:25 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/271868680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T14:21:25.847 INFO:tasks.cephadm:Setting up client nodes... 2024-03-30T14:21:25.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T14:21:26.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:25 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/271868680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T14:21:26.822 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:26 smithi031 ceph-mon[24911]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-30T14:21:27.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:26 smithi193 ceph-mon[26933]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-30T14:21:27.899 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-03-30T14:21:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDnHwhmICH1NRAAOJUi6r1bpuW++CbHm2bguQ== 2024-03-30T14:21:28.569 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-30T14:21:28.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-30T14:21:28.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T14:21:28.611 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T14:21:28.611 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-30T14:21:28.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-30T14:21:28.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:28 smithi193 ceph-mon[26933]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-30T14:21:28.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:28 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2644779884' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:21:28.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:28 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2644779884' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:21:28.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:28 smithi031 ceph-mon[24911]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-30T14:21:28.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:28 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2644779884' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:21:28.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:28 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2644779884' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:21:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:30.612 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:21:30.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:30 smithi031 ceph-mon[24911]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-30T14:21:30.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:30 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2887255237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T14:21:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:30 smithi193 ceph-mon[26933]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-30T14:21:31.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:30 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2887255237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T14:21:31.376 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":19,"active_gid":14202,"active_name":"smithi031.lkrchi","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":4026455512},{"type":"v1","addr":"172.21.15.31:6801","nonce":4026455512}]},"active_addr":"172.21.15.31:6801/4026455512","active_change":"2024-03-30T14:17:12.649404+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14220,"name":"smithi193.ikzlnb","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/","prometheus":"http://172.21.15.31:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3262072142}]},{"addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2493357632}]},{"addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3764035460}]},{"addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2626647858}]}]}} 2024-03-30T14:21:31.381 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-30T14:21:31.381 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T14:21:31.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:21:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:32.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:32 smithi031 ceph-mon[24911]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-30T14:21:32.879 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:32 smithi193 ceph-mon[26933]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-30T14:21:33.348 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:21:33.348 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:15:20.258619+0000","modified":"2024-03-30T14:21:20.252536+0000","last_up_change":"2024-03-30T14:21:16.231260+0000","last_in_change":"2024-03-30T14:20:59.009418+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:19:17.016586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"267faf1c-1984-43b9-ade7-92672d2130de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6803","nonce":636109484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6805","nonce":636109484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6809","nonce":636109484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6807","nonce":636109484}]},"public_addr":"172.21.15.31:6803/636109484","cluster_addr":"172.21.15.31:6805/636109484","heartbeat_back_addr":"172.21.15.31:6809/636109484","heartbeat_front_addr":"172.21.15.31:6807/636109484","state":["exists","up"]},{"osd":1,"uuid":"31f794ba-8f1d-4f16-ad58-1c7064b89adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6811","nonce":2328121774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6813","nonce":2328121774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6817","nonce":2328121774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6815","nonce":2328121774}]},"public_addr":"172.21.15.31:6811/2328121774","cluster_addr":"172.21.15.31:6813/2328121774","heartbeat_back_addr":"172.21.15.31:6817/2328121774","heartbeat_front_addr":"172.21.15.31:6815/2328121774","state":["exists","up"]},{"osd":2,"uuid":"60bd182a-ad38-4df8-a421-93fff90bbc5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6819","nonce":3163522635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6821","nonce":3163522635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6825","nonce":3163522635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6823","nonce":3163522635}]},"public_addr":"172.21.15.31:6819/3163522635","cluster_addr":"172.21.15.31:6821/3163522635","heartbeat_back_addr":"172.21.15.31:6825/3163522635","heartbeat_front_addr":"172.21.15.31:6823/3163522635","state":["exists","up"]},{"osd":3,"uuid":"9511f1ba-6389-4a0d-8779-6aed57c2f50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6827","nonce":1435060553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6829","nonce":1435060553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6833","nonce":1435060553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6831","nonce":1435060553}]},"public_addr":"172.21.15.31:6827/1435060553","cluster_addr":"172.21.15.31:6829/1435060553","heartbeat_back_addr":"172.21.15.31:6833/1435060553","heartbeat_front_addr":"172.21.15.31:6831/1435060553","state":["exists","up"]},{"osd":4,"uuid":"5844e23a-42e6-42ce-b357-ec1684cda09b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6801","nonce":3542342343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6803","nonce":3542342343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6807","nonce":3542342343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6805","nonce":3542342343}]},"public_addr":"172.21.15.193:6801/3542342343","cluster_addr":"172.21.15.193:6803/3542342343","heartbeat_back_addr":"172.21.15.193:6807/3542342343","heartbeat_front_addr":"172.21.15.193:6805/3542342343","state":["exists","up"]},{"osd":5,"uuid":"ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6809","nonce":71481222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6811","nonce":71481222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6815","nonce":71481222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6813","nonce":71481222}]},"public_addr":"172.21.15.193:6809/71481222","cluster_addr":"172.21.15.193:6811/71481222","heartbeat_back_addr":"172.21.15.193:6815/71481222","heartbeat_front_addr":"172.21.15.193:6813/71481222","state":["exists","up"]},{"osd":6,"uuid":"d6eb52d6-f435-49aa-a632-9b17aba3d5b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6817","nonce":3087372008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6819","nonce":3087372008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6823","nonce":3087372008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6821","nonce":3087372008}]},"public_addr":"172.21.15.193:6817/3087372008","cluster_addr":"172.21.15.193:6819/3087372008","heartbeat_back_addr":"172.21.15.193:6823/3087372008","heartbeat_front_addr":"172.21.15.193:6821/3087372008","state":["exists","up"]},{"osd":7,"uuid":"93ea6d87-04b5-4c11-92ff-0766b9788fc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6825","nonce":4193965660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6827","nonce":4193965660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6831","nonce":4193965660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6829","nonce":4193965660}]},"public_addr":"172.21.15.193:6825/4193965660","cluster_addr":"172.21.15.193:6827/4193965660","heartbeat_back_addr":"172.21.15.193:6831/4193965660","heartbeat_front_addr":"172.21.15.193:6829/4193965660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:18:14.485327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:18:45.834933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:19:14.406169+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:19:42.070138+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:03.342656+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:27.026452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:50.576227+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:21:14.289453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/4029290981":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6801/4196482206":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/4271842515":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6800/4196482206":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/2731736691":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/1332734773":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/3656726806":"2024-03-31T14:17:12.649056+0000","172.21.15.31:0/3329762819":"2024-03-31T14:15:51.576676+0000","172.21.15.31:6801/4035715893":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6801/2607887203":"2024-03-31T14:15:51.576676+0000","172.21.15.31:0/3988990777":"2024-03-31T14:16:21.262711+0000","172.21.15.31:6800/2607887203":"2024-03-31T14:15:51.576676+0000","172.21.15.31:0/3897097667":"2024-03-31T14:15:51.576676+0000","172.21.15.31:6800/4035715893":"2024-03-31T14:17:12.649056+0000","172.21.15.31:0/492591059":"2024-03-31T14:15:51.576676+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:21:34.004 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:33 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1707441586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:21:34.021 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:33 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1707441586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:21:34.031 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T14:21:34.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:21:34.253 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:35.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:34 smithi193 ceph-mon[26933]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-03-30T14:21:35.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:34 smithi031 ceph-mon[24911]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-03-30T14:21:36.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:21:36.203 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"fsid":"dc60522e-ee9f-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:15:20.258619+0000","modified":"2024-03-30T14:21:20.252536+0000","last_up_change":"2024-03-30T14:21:16.231260+0000","last_in_change":"2024-03-30T14:20:59.009418+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:19:17.016586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"267faf1c-1984-43b9-ade7-92672d2130de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6803","nonce":636109484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6805","nonce":636109484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6809","nonce":636109484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":636109484},{"type":"v1","addr":"172.21.15.31:6807","nonce":636109484}]},"public_addr":"172.21.15.31:6803/636109484","cluster_addr":"172.21.15.31:6805/636109484","heartbeat_back_addr":"172.21.15.31:6809/636109484","heartbeat_front_addr":"172.21.15.31:6807/636109484","state":["exists","up"]},{"osd":1,"uuid":"31f794ba-8f1d-4f16-ad58-1c7064b89adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6811","nonce":2328121774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6813","nonce":2328121774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6817","nonce":2328121774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2328121774},{"type":"v1","addr":"172.21.15.31:6815","nonce":2328121774}]},"public_addr":"172.21.15.31:6811/2328121774","cluster_addr":"172.21.15.31:6813/2328121774","heartbeat_back_addr":"172.21.15.31:6817/2328121774","heartbeat_front_addr":"172.21.15.31:6815/2328121774","state":["exists","up"]},{"osd":2,"uuid":"60bd182a-ad38-4df8-a421-93fff90bbc5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6819","nonce":3163522635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6821","nonce":3163522635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6825","nonce":3163522635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3163522635},{"type":"v1","addr":"172.21.15.31:6823","nonce":3163522635}]},"public_addr":"172.21.15.31:6819/3163522635","cluster_addr":"172.21.15.31:6821/3163522635","heartbeat_back_addr":"172.21.15.31:6825/3163522635","heartbeat_front_addr":"172.21.15.31:6823/3163522635","state":["exists","up"]},{"osd":3,"uuid":"9511f1ba-6389-4a0d-8779-6aed57c2f50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6827","nonce":1435060553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6829","nonce":1435060553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6833","nonce":1435060553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":1435060553},{"type":"v1","addr":"172.21.15.31:6831","nonce":1435060553}]},"public_addr":"172.21.15.31:6827/1435060553","cluster_addr":"172.21.15.31:6829/1435060553","heartbeat_back_addr":"172.21.15.31:6833/1435060553","heartbeat_front_addr":"172.21.15.31:6831/1435060553","state":["exists","up"]},{"osd":4,"uuid":"5844e23a-42e6-42ce-b357-ec1684cda09b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6801","nonce":3542342343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6803","nonce":3542342343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6807","nonce":3542342343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3542342343},{"type":"v1","addr":"172.21.15.193:6805","nonce":3542342343}]},"public_addr":"172.21.15.193:6801/3542342343","cluster_addr":"172.21.15.193:6803/3542342343","heartbeat_back_addr":"172.21.15.193:6807/3542342343","heartbeat_front_addr":"172.21.15.193:6805/3542342343","state":["exists","up"]},{"osd":5,"uuid":"ae9c89bd-e34f-4da5-9c4f-4a3e447b8bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6809","nonce":71481222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6811","nonce":71481222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6815","nonce":71481222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":71481222},{"type":"v1","addr":"172.21.15.193:6813","nonce":71481222}]},"public_addr":"172.21.15.193:6809/71481222","cluster_addr":"172.21.15.193:6811/71481222","heartbeat_back_addr":"172.21.15.193:6815/71481222","heartbeat_front_addr":"172.21.15.193:6813/71481222","state":["exists","up"]},{"osd":6,"uuid":"d6eb52d6-f435-49aa-a632-9b17aba3d5b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6817","nonce":3087372008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6819","nonce":3087372008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6823","nonce":3087372008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3087372008},{"type":"v1","addr":"172.21.15.193:6821","nonce":3087372008}]},"public_addr":"172.21.15.193:6817/3087372008","cluster_addr":"172.21.15.193:6819/3087372008","heartbeat_back_addr":"172.21.15.193:6823/3087372008","heartbeat_front_addr":"172.21.15.193:6821/3087372008","state":["exists","up"]},{"osd":7,"uuid":"93ea6d87-04b5-4c11-92ff-0766b9788fc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6825","nonce":4193965660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6827","nonce":4193965660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6831","nonce":4193965660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":4193965660},{"type":"v1","addr":"172.21.15.193:6829","nonce":4193965660}]},"public_addr":"172.21.15.193:6825/4193965660","cluster_addr":"172.21.15.193:6827/4193965660","heartbeat_back_addr":"172.21.15.193:6831/4193965660","heartbeat_front_addr":"172.21.15.193:6829/4193965660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:18:14.485327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:18:45.834933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:19:14.406169+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:19:42.070138+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:03.342656+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:27.026452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:20:50.576227+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:21:14.289453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/4029290981":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6801/4196482206":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/4271842515":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6800/4196482206":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/2731736691":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/1332734773":"2024-03-31T14:16:21.262711+0000","172.21.15.31:0/3656726806":"2024-03-31T14:17:12.649056+0000","172.21.15.31:0/3329762819":"2024-03-31T14:15:51.576676+0000","172.21.15.31:6801/4035715893":"2024-03-31T14:17:12.649056+0000","172.21.15.31:6801/2607887203":"2024-03-31T14:15:51.576676+0000","172.21.15.31:0/3988990777":"2024-03-31T14:16:21.262711+0000","172.21.15.31:6800/2607887203":"2024-03-31T14:15:51.576676+0000","172.21.15.31:0/3897097667":"2024-03-31T14:15:51.576676+0000","172.21.15.31:6800/4035715893":"2024-03-31T14:17:12.649056+0000","172.21.15.31:0/492591059":"2024-03-31T14:15:51.576676+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:21:36.842 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:36 smithi031 ceph-mon[24911]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:36.842 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:36 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3324622082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:21:36.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T14:21:36.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T14:21:36.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T14:21:36.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T14:21:36.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T14:21:36.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T14:21:36.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T14:21:36.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T14:21:37.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:36 smithi193 ceph-mon[26933]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:37.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:36 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3324622082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:21:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:37.340 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:37.344 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:37.344 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:39.030 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:38 smithi031 ceph-mon[24911]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:39.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:38 smithi193 ceph-mon[26933]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:40.774 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:40 smithi031 ceph-mon[24911]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:41.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:40 smithi193 ceph-mon[26933]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:42.917 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:42 smithi031 ceph-mon[24911]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:43.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:42 smithi193 ceph-mon[26933]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:45.025 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:44 smithi031 ceph-mon[24911]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:45.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:44 smithi193 ceph-mon[26933]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:46.812 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:46 smithi031 ceph-mon[24911]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:47.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:46 smithi193 ceph-mon[26933]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:48.750 INFO:teuthology.orchestra.run.smithi031.stdout:60129542183 2024-03-30T14:21:48.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T14:21:49.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:48 smithi193 ceph-mon[26933]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:49.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:48 smithi031 ceph-mon[24911]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:50.730 INFO:teuthology.orchestra.run.smithi031.stdout:81604378657 2024-03-30T14:21:50.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T14:21:50.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:50 smithi031 ceph-mon[24911]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:51.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:50 smithi193 ceph-mon[26933]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:51.727 INFO:teuthology.orchestra.run.smithi031.stdout:115964117020 2024-03-30T14:21:51.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T14:21:52.495 INFO:teuthology.orchestra.run.smithi031.stdout:137438953495 2024-03-30T14:21:52.496 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T14:21:52.743 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:52 smithi031 ceph-mon[24911]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:53.057 INFO:teuthology.orchestra.run.smithi031.stdout:206158430222 2024-03-30T14:21:53.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T14:21:53.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:52 smithi193 ceph-mon[26933]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:53.451 INFO:teuthology.orchestra.run.smithi031.stdout:240518168584 2024-03-30T14:21:53.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T14:21:54.338 INFO:teuthology.orchestra.run.smithi031.stdout:38654705710 2024-03-30T14:21:54.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T14:21:54.343 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:54.774 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:54.775 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:54.775 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:54.776 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:54.776 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:54.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:54 smithi031 ceph-mon[24911]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:55.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:54 smithi193 ceph-mon[26933]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:55.281 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:21:56.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:56 smithi031 ceph-mon[24911]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:57.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:56 smithi193 ceph-mon[26933]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:58.905 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:21:58 smithi031 ceph-mon[24911]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:21:59.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:21:58 smithi193 ceph-mon[26933]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:00.785 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:00 smithi031 ceph-mon[24911]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:01.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:00 smithi193 ceph-mon[26933]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:02.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:02 smithi031 ceph-mon[24911]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:02.871 INFO:teuthology.orchestra.run.smithi031.stdout:38654705712 2024-03-30T14:22:03.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:02 smithi193 ceph-mon[26933]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:03.516 INFO:teuthology.orchestra.run.smithi031.stdout:137438953497 2024-03-30T14:22:03.599 INFO:teuthology.orchestra.run.smithi031.stdout:240518168586 2024-03-30T14:22:04.013 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:03 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2880978499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:22:04.014 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:03 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/47215405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:22:04.014 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:03 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1357374921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:22:04.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:03 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2880978499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:22:04.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:03 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/47215405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:22:04.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:03 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1357374921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:22:04.287 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168584 got 240518168586 for osd.7 2024-03-30T14:22:04.287 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:04.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705710 got 38654705712 for osd.0 2024-03-30T14:22:04.685 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:05.009 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:04 smithi031 ceph-mon[24911]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:05.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:04 smithi193 ceph-mon[26933]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:05.089 INFO:teuthology.orchestra.run.smithi031.stdout:81604378660 2024-03-30T14:22:05.305 INFO:teuthology.orchestra.run.smithi031.stdout:60129542186 2024-03-30T14:22:05.737 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953495 got 137438953497 for osd.4 2024-03-30T14:22:05.738 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:05.907 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:05 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/111161669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:22:05.907 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:05 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3225934661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:22:06.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:05 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/111161669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:22:06.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:05 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3225934661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:22:06.106 INFO:teuthology.orchestra.run.smithi031.stdout:206158430226 2024-03-30T14:22:06.323 INFO:teuthology.orchestra.run.smithi031.stdout:171798691860 2024-03-30T14:22:06.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T14:22:07.075 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:06 smithi031 ceph-mon[24911]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:07.075 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:06 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3093204490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:22:07.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:06 smithi193 ceph-mon[26933]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:07.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:06 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3093204490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:22:07.094 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378657 got 81604378660 for osd.2 2024-03-30T14:22:07.095 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:07.658 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430226 for osd.6 2024-03-30T14:22:07.658 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:08.311 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:22:08.731 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-03-30T14:22:08.732 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:08.969 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:08 smithi031 ceph-mon[24911]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:09.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:08 smithi193 ceph-mon[26933]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:09.159 INFO:teuthology.orchestra.run.smithi031.stdout:115964117024 2024-03-30T14:22:10.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:09 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2845988082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:22:10.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:09 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2845988082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:22:10.799 INFO:teuthology.orchestra.run.smithi031.stdout:171798691863 2024-03-30T14:22:10.999 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:10 smithi031 ceph-mon[24911]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:11.023 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117020 got 115964117024 for osd.3 2024-03-30T14:22:11.023 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:11.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:10 smithi193 ceph-mon[26933]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:11.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691860 got 171798691863 for osd.5 2024-03-30T14:22:11.681 DEBUG:teuthology.parallel:result is None 2024-03-30T14:22:11.681 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T14:22:11.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:22:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:22:12.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:11 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/707130945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:22:12.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:11 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/707130945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:22:13.071 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:12 smithi031 ceph-mon[24911]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:13.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:12 smithi193 ceph-mon[26933]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:13.788 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:22:13.789 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-30T14:22:14.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:22:14.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:22:14.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:22:14.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:22:14.412 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-30T14:22:12.754453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49008,"kb_used_data":6384,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912208,"statfs":{"total":767960285184,"available":767910100992,"internally_reserved":0,"allocated":6537216,"data_stored":3571850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001873"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":222,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-30T14:21:20.259875+0000","last_change":"2024-03-30T14:21:20.259875+0000","last_active":"2024-03-30T14:21:20.259875+0000","last_peered":"2024-03-30T14:21:20.259875+0000","last_clean":"2024-03-30T14:21:20.259875+0000","last_became_active":"2024-03-30T14:21:20.259349+0000","last_became_peered":"2024-03-30T14:21:20.259349+0000","last_unstale":"2024-03-30T14:21:20.259875+0000","last_undegraded":"2024-03-30T14:21:20.259875+0000","last_fullsized":"2024-03-30T14:21:20.259875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:19:17.659659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:19:17.659659+0000","last_clean_scrub_stamp":"2024-03-30T14:19:17.659659+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:38:28.807932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":1146880,"data_stored":776819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.622},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.79200000000000004}]},{"osd":1,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.72599999999999998}]},{"osd":3,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60299999999999998}]},{"osd":4,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.376},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41999999999999998}]},{"osd":5,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.50900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.33600000000000002,"15min":0.314},"min":{"1min":0.25,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.312},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.34000000000000002,"15min":0.32100000000000001},"min":{"1min":0.25700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.53900000000000003,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36899999999999999}]},{"osd":2,"last update":"Sat Mar 30 14:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36899999999999999,"15min":0.36099999999999999},"min":{"1min":0.28599999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.38100000000000001,"15min":0.372},"min":{"1min":0.26800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Mar 30 14:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Mar 30 14:22:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80600000000000005,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.437,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79500000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.66900000000000004}]},{"osd":5,"last update":"Sat Mar 30 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":14.877000000000001,"5min":14.877000000000001,"15min":14.877000000000001},"last":0.68400000000000005}]},{"osd":6,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.624}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 14:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.317},"min":{"1min":0.251,"5min":0.214,"15min":0.214},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.36699999999999999,"15min":0.33600000000000002},"min":{"1min":0.28299999999999997,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.439}]},{"osd":2,"last update":"Sat Mar 30 14:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.38300000000000001,"15min":0.373},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.35899999999999999},"min":{"1min":0.317,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Mar 30 14:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.91200000000000003,"15min":0.97699999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.028,"5min":11.472,"15min":11.472},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.91600000000000004,"15min":0.98899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.94599999999999995,"5min":11.615,"15min":11.615},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Mar 30 14:21:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.68100000000000005}]},{"osd":5,"last update":"Sat Mar 30 14:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.73899999999999999}]},{"osd":6,"last update":"Sat Mar 30 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6308,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738844,"statfs":{"total":95995035648,"available":95988576256,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"min":{"1min":0.318,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.34599999999999997}]},{"osd":1,"last update":"Sat Mar 30 14:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.374},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.48399999999999999,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.35499999999999998},"min":{"1min":0.32300000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.50800000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.247}]},{"osd":3,"last update":"Sat Mar 30 14:21:44 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42399999999999999,"15min":0.42099999999999999},"min":{"1min":0.34200000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51700000000000002,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.31},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.47599999999999998},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63400000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57899999999999996,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82999999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84099999999999997,"5min":0.875,"15min":0.875},"last":0.49099999999999999}]},{"osd":5,"last update":"Sat Mar 30 14:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42699999999999999}]},{"osd":6,"last update":"Sat Mar 30 14:21:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.442},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39900000000000002,"15min":0.39700000000000002},"min":{"1min":0.29999999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57699999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Mar 30 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.28499999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56599999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.34100000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Mar 30 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.502},"min":{"1min":0.379,"5min":0.317,"15min":0.317},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.499,"15min":0.499},"min":{"1min":0.36199999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.51000000000000001}]},{"osd":4,"last update":"Sat Mar 30 14:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.625,"15min":0.61299999999999999},"min":{"1min":0.48299999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.75}]},{"osd":5,"last update":"Sat Mar 30 14:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Mar 30 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.75800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5796,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739356,"statfs":{"total":95995035648,"available":95989100544,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53200000000000003,"15min":0.52300000000000002},"min":{"1min":0.45100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":0.67100000000000004,"15min":0.56999999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":14.766,"5min":14.766,"15min":14.766},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.42999999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54200000000000004}]},{"osd":2,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.66000000000000003}]},{"osd":3,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.2889999999999999,"5min":0.67800000000000005,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":14.749000000000001,"5min":14.749000000000001,"15min":14.749000000000001},"last":0.625}]},{"osd":5,"last update":"Sat Mar 30 14:21:32 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.64500000000000002}]},{"osd":6,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":5,"up_from":40,"seq":171798691864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.67500000000000004}]},{"osd":4,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57699999999999996}]},{"osd":6,"last update":"Sat Mar 30 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:22:14.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:22:14.637 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:22:14.972 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:14 smithi031 ceph-mon[24911]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:15.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:14 smithi193 ceph-mon[26933]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:15.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:15 smithi031 ceph-mon[24911]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:16.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:15 smithi193 ceph-mon[26933]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:16.439 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:22:16.441 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-30T14:22:17.025 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:16 smithi031 ceph-mon[24911]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:17.049 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-30T14:22:14.754967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49008,"kb_used_data":6384,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912208,"statfs":{"total":767960285184,"available":767910100992,"internally_reserved":0,"allocated":6537216,"data_stored":3571850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002032"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":222,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-30T14:21:20.259875+0000","last_change":"2024-03-30T14:21:20.259875+0000","last_active":"2024-03-30T14:21:20.259875+0000","last_peered":"2024-03-30T14:21:20.259875+0000","last_clean":"2024-03-30T14:21:20.259875+0000","last_became_active":"2024-03-30T14:21:20.259349+0000","last_became_peered":"2024-03-30T14:21:20.259349+0000","last_unstale":"2024-03-30T14:21:20.259875+0000","last_undegraded":"2024-03-30T14:21:20.259875+0000","last_fullsized":"2024-03-30T14:21:20.259875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:19:17.659659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:19:17.659659+0000","last_clean_scrub_stamp":"2024-03-30T14:19:17.659659+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:38:28.807932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":1146880,"data_stored":776819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":6,"up_from":48,"seq":206158430228,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.62},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.628}]},{"osd":3,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.66600000000000004}]},{"osd":4,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.51000000000000001}]},{"osd":5,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.33600000000000002,"15min":0.314},"min":{"1min":0.25,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.34000000000000002,"15min":0.32100000000000001},"min":{"1min":0.25700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.53900000000000003,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Mar 30 14:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36899999999999999,"15min":0.36099999999999999},"min":{"1min":0.28599999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.38100000000000001,"15min":0.372},"min":{"1min":0.26800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Mar 30 14:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.37},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Mar 30 14:22:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80600000000000005,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.437,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79500000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Mar 30 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":14.877000000000001,"5min":14.877000000000001,"15min":14.877000000000001},"last":0.57699999999999996}]},{"osd":6,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.436},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.505}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 14:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.317},"min":{"1min":0.251,"5min":0.214,"15min":0.214},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.36699999999999999,"15min":0.33600000000000002},"min":{"1min":0.28299999999999997,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.439}]},{"osd":2,"last update":"Sat Mar 30 14:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.38300000000000001,"15min":0.373},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.35899999999999999},"min":{"1min":0.317,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Mar 30 14:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.91200000000000003,"15min":0.97699999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.028,"5min":11.472,"15min":11.472},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.91600000000000004,"15min":0.98899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.94599999999999995,"5min":11.615,"15min":11.615},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Mar 30 14:21:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.68100000000000005}]},{"osd":5,"last update":"Sat Mar 30 14:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.73899999999999999}]},{"osd":6,"last update":"Sat Mar 30 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6308,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738844,"statfs":{"total":95995035648,"available":95988576256,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"min":{"1min":0.318,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.34599999999999997}]},{"osd":1,"last update":"Sat Mar 30 14:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.374},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.48399999999999999,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.35499999999999998},"min":{"1min":0.32300000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.50800000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.247}]},{"osd":3,"last update":"Sat Mar 30 14:21:44 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42399999999999999,"15min":0.42099999999999999},"min":{"1min":0.34200000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51700000000000002,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.31},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.47599999999999998},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63400000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57899999999999996,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82999999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84099999999999997,"5min":0.875,"15min":0.875},"last":0.49099999999999999}]},{"osd":5,"last update":"Sat Mar 30 14:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42699999999999999}]},{"osd":6,"last update":"Sat Mar 30 14:21:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.442},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39900000000000002,"15min":0.39700000000000002},"min":{"1min":0.29999999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57699999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38}]},{"osd":1,"last update":"Sat Mar 30 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.28499999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56599999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.34100000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Mar 30 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.502},"min":{"1min":0.379,"5min":0.317,"15min":0.317},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.499,"15min":0.499},"min":{"1min":0.36199999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.45700000000000002}]},{"osd":4,"last update":"Sat Mar 30 14:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.625,"15min":0.61299999999999999},"min":{"1min":0.48299999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Mar 30 14:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.75700000000000001}]},{"osd":6,"last update":"Sat Mar 30 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5796,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739356,"statfs":{"total":95995035648,"available":95989100544,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53200000000000003,"15min":0.52300000000000002},"min":{"1min":0.45100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":0.67100000000000004,"15min":0.56999999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":14.766,"5min":14.766,"15min":14.766},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.42999999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54200000000000004}]},{"osd":2,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.66000000000000003}]},{"osd":3,"last update":"Sat Mar 30 14:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.2889999999999999,"5min":0.67800000000000005,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":14.749000000000001,"5min":14.749000000000001,"15min":14.749000000000001},"last":0.625}]},{"osd":5,"last update":"Sat Mar 30 14:21:32 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.64500000000000002}]},{"osd":6,"last update":"Sat Mar 30 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":5,"up_from":40,"seq":171798691864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.67500000000000004}]},{"osd":4,"last update":"Sat Mar 30 14:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57699999999999996}]},{"osd":6,"last update":"Sat Mar 30 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:22:17.051 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T14:22:17.051 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T14:22:17.051 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-30T14:22:17.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-30T14:22:17.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:16 smithi193 ceph-mon[26933]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/mon.smithi031/config 2024-03-30T14:22:18.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:17 smithi193 ceph-mon[26933]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:18.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:17 smithi031 ceph-mon[24911]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:18.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:18 smithi031 ceph-mon[24911]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:18.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:18.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:18 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:19.048 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:22:19.048 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T14:22:19.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:18 smithi193 ceph-mon[26933]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:19.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:19.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:19.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:19.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:18 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:19.680 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-30T14:22:19.680 INFO:tasks.cephadm:Setup complete, yielding 2024-03-30T14:22:19.681 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T14:22:19.692 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:22:19.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-03-30T14:22:20.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:19 smithi031 ceph-mon[24911]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:20.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:19 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1586575890' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T14:22:20.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:19 smithi193 ceph-mon[26933]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:20.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:19 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1586575890' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T14:22:21.639 INFO:teuthology.orchestra.run.smithi031.stdout:Backend: cephadm 2024-03-30T14:22:21.639 INFO:teuthology.orchestra.run.smithi031.stdout:Available: Yes 2024-03-30T14:22:21.639 INFO:teuthology.orchestra.run.smithi031.stdout:Paused: No 2024-03-30T14:22:21.940 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:21 smithi031 ceph-mon[24911]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:21.941 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:21 smithi031 ceph-mon[24911]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:22.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:21 smithi193 ceph-mon[26933]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:22.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:21 smithi193 ceph-mon[26933]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:22.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-03-30T14:22:23.840 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:23 smithi031 ceph-mon[24911]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:24.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:23 smithi193 ceph-mon[26933]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:24.277 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 9fc4502dff1a 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (5m) 2m ago 5m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (5m) 69s ago 5m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (3m) 2m ago 5m 45.1M - 8.3.5 dad864ee21e9 c132179ebde6 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 330ec216c990 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (4m) 69s ago 4m 412M - 17.2.0 e1d6a67b021e def8a6a77a9b 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (7m) 2m ago 7m 42.1M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (4m) 69s ago 4m 40.2M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (5m) 2m ago 5m 17.3M - 1dbe0e931976 a5b6d8e0efa4 2024-03-30T14:22:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (4m) 69s ago 4m 17.6M - 1dbe0e931976 3f7d1e5ad9be 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (4m) 2m ago 4m 33.1M 3217M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (3m) 2m ago 3m 34.5M 3217M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (3m) 2m ago 3m 33.0M 3217M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (2m) 2m ago 2m 25.7M 3217M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (2m) 69s ago 2m 36.0M 3984M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (2m) 69s ago 2m 32.5M 3984M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (98s) 69s ago 97s 31.3M 3984M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (74s) 69s ago 74s 11.3M 3984M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:22:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (4m) 2m ago 4m 55.6M - 514e6a882f6e 15161d5c677a 2024-03-30T14:22:24.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-03-30T14:22:25.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:25 smithi031 ceph-mon[24911]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:26.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:25 smithi193 ceph-mon[26933]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-30T14:22:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-30T14:22:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:crash 2/2 2m ago 6m * 2024-03-30T14:22:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-30T14:22:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 2/2 2m ago 6m count:2 2024-03-30T14:22:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:mon 2/2 2m ago 5m smithi031:172.21.15.31=smithi031;smithi193:172.21.15.193=smithi193;count:2 2024-03-30T14:22:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-30T14:22:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:osd 8 2m ago - 2024-03-30T14:22:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-30T14:22:27.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:26 smithi193 ceph-mon[26933]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:27.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:26 smithi031 ceph-mon[24911]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:27.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-03-30T14:22:28.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:28 smithi031 ceph-mon[24911]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:28.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:28 smithi031 ceph-mon[24911]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:29.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:28 smithi193 ceph-mon[26933]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:29.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:28 smithi193 ceph-mon[26933]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:29.275 INFO:teuthology.orchestra.run.smithi031.stdout:HOST ADDR LABELS STATUS 2024-03-30T14:22:29.275 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 172.21.15.31 2024-03-30T14:22:29.275 INFO:teuthology.orchestra.run.smithi031.stdout:smithi193 172.21.15.193 2024-03-30T14:22:29.275 INFO:teuthology.orchestra.run.smithi031.stdout:2 hosts in cluster 2024-03-30T14:22:29.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-03-30T14:22:30.841 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:30 smithi031 ceph-mon[24911]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:30.841 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:30 smithi031 ceph-mon[24911]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:31.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:30 smithi193 ceph-mon[26933]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:31.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:30 smithi193 ceph-mon[26933]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:31.770 INFO:teuthology.orchestra.run.smithi031.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-03-30T14:22:31.770 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 400G LVM detected, locked 2024-03-30T14:22:31.771 INFO:teuthology.orchestra.run.smithi031.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 400G LVM detected, locked 2024-03-30T14:22:32.383 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-30T14:22:32.392 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:22:32.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-30T14:22:32.423 INFO:teuthology.orchestra.run.smithi031.stderr:+ systemctl stop nfs-server 2024-03-30T14:22:32.430 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi193.front.sepia.ceph.com 2024-03-30T14:22:32.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-30T14:22:32.462 INFO:teuthology.orchestra.run.smithi193.stderr:+ systemctl stop nfs-server 2024-03-30T14:22:32.471 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T14:22:32.480 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:22:32.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs volume create foofs' 2024-03-30T14:22:32.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:32 smithi193 ceph-mon[26933]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:32.983 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:32 smithi031 ceph-mon[24911]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:34.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:33 smithi031 ceph-mon[24911]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:34.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:33 smithi193 ceph-mon[26933]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:35.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:34 smithi031 ceph-mon[24911]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:35.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:34 smithi193 ceph-mon[26933]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:36.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:35 smithi031 ceph-mon[24911]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:35 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-30T14:22:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:35 smithi031 ceph-mon[24911]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:35 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-30T14:22:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:35 smithi031 ceph-mon[24911]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T14:22:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:35 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-30T14:22:36.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:35 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[24907]: 2024-03-30T14:22:35.893+0000 7f606baff700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:22:36.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:35 smithi193 ceph-mon[26933]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:36.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:35 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-30T14:22:36.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:35 smithi193 ceph-mon[26933]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:36.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:35 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-30T14:22:36.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:35 smithi193 ceph-mon[26933]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T14:22:36.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:35 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-30T14:22:36.596 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-03-30T14:22:36.605 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-03-30T14:22:36.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: fsmap foofs:0 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: Saving service mds.foofs spec with placement count:2 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:37.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:22:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T14:22:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:37.106 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:36 smithi031 ceph-mon[24911]: Deploying daemon mds.foofs.smithi031.xsynqy on smithi031 2024-03-30T14:22:37.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-30T14:22:37.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T14:22:37.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: fsmap foofs:0 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: Saving service mds.foofs spec with placement count:2 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:37.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:36 smithi193 ceph-mon[26933]: Deploying daemon mds.foofs.smithi031.xsynqy on smithi031 2024-03-30T14:22:38.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:37 smithi031 ceph-mon[24911]: pgmap v213: 65 pgs: 4 creating+activating, 12 creating+peering, 35 unknown, 14 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:38.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:37 smithi031 ceph-mon[24911]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T14:22:38.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:37 smithi193 ceph-mon[26933]: pgmap v213: 65 pgs: 4 creating+activating, 12 creating+peering, 35 unknown, 14 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:38.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:37 smithi193 ceph-mon[26933]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T14:22:39.136 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:38 smithi031 ceph-mon[24911]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T14:22:39.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:38 smithi193 ceph-mon[26933]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T14:22:39.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:22:39.825 INFO:teuthology.orchestra.run.smithi031.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-30T14:16:10.687846Z", "last_refresh": "2024-03-30T14:19:52.160735Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-30T14:16:04.765473Z", "last_refresh": "2024-03-30T14:19:52.160802Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-30T14:16:07.753420Z", "last_refresh": "2024-03-30T14:19:52.160868Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-30T14:22:35.915295Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-30T14:22:35.908714Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-30T14:16:03.520237Z", "last_refresh": "2024-03-30T14:19:52.160659Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi031:172.21.15.31=smithi031", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-30T14:16:53.746502Z", "last_refresh": "2024-03-30T14:19:52.160514Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-30T14:16:09.261132Z", "last_refresh": "2024-03-30T14:19:52.160932Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-30T14:19:52.161071Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-30T14:16:06.520988Z", "last_refresh": "2024-03-30T14:19:52.160995Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-30T14:22:40.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:39 smithi031 ceph-mon[24911]: pgmap v216: 65 pgs: 4 creating+activating, 12 creating+peering, 28 unknown, 21 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:40.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:39 smithi193 ceph-mon[26933]: pgmap v216: 65 pgs: 4 creating+activating, 12 creating+peering, 28 unknown, 21 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:40.814 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-30T14:22:41.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: Deploying daemon mds.foofs.smithi193.shgxxs on smithi193 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: mds.? [v2:172.21.15.31:6834/669631789,v1:172.21.15.31:6835/669631789] up:boot 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: daemon mds.foofs.smithi031.xsynqy assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: Cluster is now healthy 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: fsmap foofs:0 1 up:standby 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:creating} 2024-03-30T14:22:41.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:41 smithi031 ceph-mon[24911]: daemon mds.foofs.smithi031.xsynqy is now active in filesystem foofs as rank 0 2024-03-30T14:22:41.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:41.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:41.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: Deploying daemon mds.foofs.smithi193.shgxxs on smithi193 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: mds.? [v2:172.21.15.31:6834/669631789,v1:172.21.15.31:6835/669631789] up:boot 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: daemon mds.foofs.smithi031.xsynqy assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: Cluster is now healthy 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: fsmap foofs:0 1 up:standby 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:creating} 2024-03-30T14:22:41.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:41 smithi193 ceph-mon[26933]: daemon mds.foofs.smithi031.xsynqy is now active in filesystem foofs as rank 0 2024-03-30T14:22:41.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-03-30T14:22:42.452 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:42 smithi031 ceph-mon[24911]: pgmap v217: 65 pgs: 4 creating+activating, 12 creating+peering, 49 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:42.452 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:42 smithi031 ceph-mon[24911]: mds.? [v2:172.21.15.31:6834/669631789,v1:172.21.15.31:6835/669631789] up:active 2024-03-30T14:22:42.452 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:42 smithi031 ceph-mon[24911]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 2024-03-30T14:22:42.646 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:42 smithi193 ceph-mon[26933]: pgmap v217: 65 pgs: 4 creating+activating, 12 creating+peering, 49 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:22:42.646 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:42 smithi193 ceph-mon[26933]: mds.? [v2:172.21.15.31:6834/669631789,v1:172.21.15.31:6835/669631789] up:active 2024-03-30T14:22:42.646 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:42 smithi193 ceph-mon[26933]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 2024-03-30T14:22:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:22:44.043 INFO:teuthology.orchestra.run.smithi031.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-30T14:16:10.687846Z", "last_refresh": "2024-03-30T14:19:52.160735Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-30T14:16:04.765473Z", "last_refresh": "2024-03-30T14:19:52.160802Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-30T14:16:07.753420Z", "last_refresh": "2024-03-30T14:19:52.160868Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-30T14:22:35.915295Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-30T14:22:35.908714Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-30T14:16:03.520237Z", "last_refresh": "2024-03-30T14:19:52.160659Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi031:172.21.15.31=smithi031", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-30T14:16:53.746502Z", "last_refresh": "2024-03-30T14:19:52.160514Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-30T14:16:09.261132Z", "last_refresh": "2024-03-30T14:19:52.160932Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-30T14:19:52.161071Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-30T14:16:06.520988Z", "last_refresh": "2024-03-30T14:19:52.160995Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-30T14:22:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-30T14:22:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:boot 2024-03-30T14:22:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:22:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:22:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:43 smithi193 ceph-mon[26933]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:22:44.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-30T14:22:44.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:44.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:44.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:44.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:44.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:boot 2024-03-30T14:22:44.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:22:44.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:22:44.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:43 smithi031 ceph-mon[24911]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:22:44.845 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-30T14:22:45.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:44 smithi031 ceph-mon[24911]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:45.211 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:44 smithi193 ceph-mon[26933]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:45.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-03-30T14:22:46.242 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:45 smithi031 ceph-mon[24911]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-30T14:22:46.242 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:45 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:46.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:45 smithi193 ceph-mon[26933]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-30T14:22:46.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:45 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:47.947 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:22:47.947 INFO:teuthology.orchestra.run.smithi031.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-30T14:16:10.687846Z", "last_refresh": "2024-03-30T14:19:52.160735Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-30T14:16:04.765473Z", "last_refresh": "2024-03-30T14:19:52.160802Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-30T14:16:07.753420Z", "last_refresh": "2024-03-30T14:19:52.160868Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-30T14:22:35.915295Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-30T14:22:35.908714Z", "last_refresh": "2024-03-30T14:22:44.988017Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-30T14:16:03.520237Z", "last_refresh": "2024-03-30T14:19:52.160659Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi031:172.21.15.31=smithi031", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-30T14:16:53.746502Z", "last_refresh": "2024-03-30T14:19:52.160514Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-30T14:16:09.261132Z", "last_refresh": "2024-03-30T14:19:52.160932Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-30T14:19:52.161071Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-30T14:16:06.520988Z", "last_refresh": "2024-03-30T14:19:52.160995Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-30T14:22:48.239 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:47 smithi031 ceph-mon[24911]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-30T14:22:48.239 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:47 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:48.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:47 smithi193 ceph-mon[26933]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-30T14:22:48.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:47 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:49.257 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:48 smithi031 ceph-mon[24911]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:49.280 INFO:tasks.cephadm:mds.foofs has 1/2 2024-03-30T14:22:49.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:48 smithi193 ceph-mon[26933]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:50.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-03-30T14:22:50.304 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:50 smithi031 ceph-mon[24911]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-30T14:22:50.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:50 smithi193 ceph-mon[26933]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-30T14:22:52.125 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:52 smithi031 ceph-mon[24911]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-30T14:22:52.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:52 smithi193 ceph-mon[26933]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-30T14:22:52.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:22:52.553 INFO:teuthology.orchestra.run.smithi031.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-30T14:16:10.687846Z", "last_refresh": "2024-03-30T14:22:52.451027Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-30T14:16:04.765473Z", "last_refresh": "2024-03-30T14:22:44.987292Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-30T14:16:07.753420Z", "last_refresh": "2024-03-30T14:22:52.451167Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-30T14:22:35.915295Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-30T14:22:35.908714Z", "last_refresh": "2024-03-30T14:22:44.988017Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-30T14:16:03.520237Z", "last_refresh": "2024-03-30T14:22:44.987463Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi031:172.21.15.31=smithi031", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-30T14:16:53.746502Z", "last_refresh": "2024-03-30T14:22:44.987552Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-30T14:16:09.261132Z", "last_refresh": "2024-03-30T14:22:44.987632Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-30T14:22:44.987709Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-30T14:16:06.520988Z", "last_refresh": "2024-03-30T14:22:52.451296Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-30T14:22:53.323 INFO:tasks.cephadm:mds.foofs has 2/2 2024-03-30T14:22:53.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T14:22:53.333 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:22:53.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-03-30T14:22:53.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:53 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:53.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:53 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:53.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:53 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:53.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:53 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:53.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:53 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:53.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:53 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:53.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:53 smithi031 ceph-mon[24911]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:53.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:53 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:53.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:53 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:53.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:53 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:53.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:53 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:53.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:53 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:53.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:53 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:53.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:53 smithi193 ceph-mon[26933]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:22:54.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:54 smithi193 ceph-mon[26933]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-30T14:22:54.717 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:54 smithi031 ceph-mon[24911]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-30T14:22:56.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:55 smithi031 ceph-mon[24911]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 426 B/s wr, 1 op/s 2024-03-30T14:22:56.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:55 smithi031 ceph-mon[24911]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:56.105 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:55 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-30T14:22:56.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:55 smithi193 ceph-mon[26933]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 426 B/s wr, 1 op/s 2024-03-30T14:22:56.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:55 smithi193 ceph-mon[26933]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:22:56.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:55 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-30T14:22:56.874 INFO:teuthology.orchestra.run.smithi031.stdout:NFS Cluster Created Successfully 2024-03-30T14:22:57.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:56 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-30T14:22:57.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:56 smithi031 ceph-mon[24911]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T14:22:57.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:56 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-30T14:22:57.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:56 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-30T14:22:57.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:56 smithi193 ceph-mon[26933]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T14:22:57.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:56 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-30T14:22:57.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-03-30T14:22:58.096 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: pgmap v226: 97 pgs: 15 creating+peering, 17 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: Saving service nfs.foo spec with placement count:2 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-30T14:22:58.097 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: Rados config object exists: conf-nfs.foo 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh-rgw 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:58.098 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:57 smithi031 ceph-mon[24911]: Deploying daemon nfs.foo.0.0.smithi031.juvmqh on smithi031 2024-03-30T14:22:58.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: pgmap v226: 97 pgs: 15 creating+peering, 17 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: Saving service nfs.foo spec with placement count:2 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:22:58.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: Rados config object exists: conf-nfs.foo 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh-rgw 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:58.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:57 smithi193 ceph-mon[26933]: Deploying daemon nfs.foo.0.0.smithi031.juvmqh on smithi031 2024-03-30T14:22:58.935 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:58 smithi031 ceph-mon[24911]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T14:22:59.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:58 smithi193 ceph-mon[26933]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T14:22:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:Invalid command: Unexpected argument '--clusterid' 2024-03-30T14:22:59.649 INFO:teuthology.orchestra.run.smithi031.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-03-30T14:22:59.649 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-03-30T14:22:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:22:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "bind": "/fake", 2024-03-30T14:22:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "fs": "foofs", 2024-03-30T14:22:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-30T14:22:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster": "foo", 2024-03-30T14:22:59.998 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "RW" 2024-03-30T14:22:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:23:00.090 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:59 smithi031 ceph-mon[24911]: pgmap v229: 97 pgs: 15 creating+peering, 14 unknown, 68 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:23:00.090 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:22:59 smithi031 ceph-mon[24911]: mgrmap e20: smithi031.lkrchi(active, since 5m), standbys: smithi193.ikzlnb 2024-03-30T14:23:00.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:59 smithi193 ceph-mon[26933]: pgmap v229: 97 pgs: 15 creating+peering, 14 unknown, 68 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:23:00.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:22:59 smithi193 ceph-mon[26933]: mgrmap e20: smithi031.lkrchi(active, since 5m), standbys: smithi193.ikzlnb 2024-03-30T14:23:01.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:00 smithi031 ceph-mon[24911]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:01.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-30T14:23:01.104 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:00 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-30T14:23:01.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:00 smithi193 ceph-mon[26933]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:01.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-30T14:23:01.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:00 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-30T14:23:01.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-03-30T14:23:02.238 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:02 smithi031 ceph-mon[24911]: pgmap v230: 97 pgs: 15 creating+peering, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-30T14:23:02.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:02 smithi193 ceph-mon[26933]: pgmap v230: 97 pgs: 15 creating+peering, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: Creating key for client.nfs.foo.1.0.smithi193.odcsji 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:23:03.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:03 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:03.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: Creating key for client.nfs.foo.1.0.smithi193.odcsji 2024-03-30T14:23:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:23:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-30T14:23:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-30T14:23:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:23:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:23:03.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:03 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:04.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:04 smithi193 ceph-mon[26933]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 148 B/s rd, 296 B/s wr, 0 op/s 2024-03-30T14:23:04.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:04 smithi031 ceph-mon[24911]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 148 B/s rd, 296 B/s wr, 0 op/s 2024-03-30T14:23:05.327 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:05 smithi193 ceph-mon[26933]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:23:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:23:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:23:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T14:23:05.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:05 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:05.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:05 smithi031 ceph-mon[24911]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:05.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:23:05.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:23:05.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:23:05.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T14:23:05.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:05 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:06.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:06 smithi031 ceph-mon[24911]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-30T14:23:06.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:06 smithi031 ceph-mon[24911]: Rados config object exists: conf-nfs.foo 2024-03-30T14:23:06.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:06 smithi031 ceph-mon[24911]: Creating key for client.nfs.foo.1.0.smithi193.odcsji-rgw 2024-03-30T14:23:06.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:06 smithi031 ceph-mon[24911]: Deploying daemon nfs.foo.1.0.smithi193.odcsji on smithi193 2024-03-30T14:23:06.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:06 smithi031 ceph-mon[24911]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:06.381 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:06 smithi193 ceph-mon[26933]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-30T14:23:06.381 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:06 smithi193 ceph-mon[26933]: Rados config object exists: conf-nfs.foo 2024-03-30T14:23:06.382 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:06 smithi193 ceph-mon[26933]: Creating key for client.nfs.foo.1.0.smithi193.odcsji-rgw 2024-03-30T14:23:06.382 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:06 smithi193 ceph-mon[26933]: Deploying daemon nfs.foo.1.0.smithi193.odcsji on smithi193 2024-03-30T14:23:06.382 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:06 smithi193 ceph-mon[26933]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:07.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:07 smithi031 ceph-mon[24911]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:07.380 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:07 smithi193 ceph-mon[26933]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:08.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:08 smithi031 ceph-mon[24911]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-30T14:23:08.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:08 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:08.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:08 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:08.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:08 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:08.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:08 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:08.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:08 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:08.699 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:08 smithi193 ceph-mon[26933]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-30T14:23:08.699 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:08 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:08.699 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:08 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:08.699 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:08 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:08.700 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:08 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:08.700 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:08 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:09.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:09 smithi031 ceph-mon[24911]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:09.692 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:09 smithi193 ceph-mon[26933]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:10.371 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:10 smithi031 ceph-mon[24911]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T14:23:10.371 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:10 smithi031 ceph-mon[24911]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:10.371 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:10 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:10.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:10 smithi193 ceph-mon[26933]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T14:23:10.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:10 smithi193 ceph-mon[26933]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:10.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:10 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:11.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:11 smithi031 ceph-mon[24911]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:11.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:11 smithi193 ceph-mon[26933]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:12.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-30T14:23:12.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:12 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-30T14:23:12.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:12.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:12.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:12.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:12.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:12 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:13.139 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo ?:2049 2/2 3s ago 16s count:2 2024-03-30T14:23:13.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:13 smithi031 ceph-mon[24911]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-30T14:23:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T14:23:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-30T14:23:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-30T14:23:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:23:13.355 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:23:13.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:13 smithi193 ceph-mon[26933]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:13.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-30T14:23:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T14:23:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-30T14:23:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-30T14:23:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:23:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:23:14.060 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-30T14:23:14.070 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:23:14.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-03-30T14:23:14.101 INFO:teuthology.orchestra.run.smithi031.stderr:+ mkdir /mnt/foo 2024-03-30T14:23:14.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-03-30T14:23:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:++ hostname 2024-03-30T14:23:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:+ mount -t nfs smithi031:/fake /mnt/foo -o sync 2024-03-30T14:23:14.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-03-30T14:23:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:14 smithi193 ceph-mon[26933]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-30T14:23:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:14 smithi193 ceph-mon[26933]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-30T14:23:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-30T14:23:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-30T14:23:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-30T14:23:14.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:14 smithi193 ceph-mon[26933]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T14:23:14.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:14 smithi031 ceph-mon[24911]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-30T14:23:14.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:14 smithi031 ceph-mon[24911]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:14.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-30T14:23:14.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-30T14:23:14.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-30T14:23:14.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-30T14:23:14.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:14 smithi031 ceph-mon[24911]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T14:23:14.630 INFO:teuthology.orchestra.run.smithi031.stderr:+ echo test 2024-03-30T14:23:14.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-03-30T14:23:14.729 INFO:teuthology.orchestra.run.smithi031.stderr:+ sync 2024-03-30T14:23:14.891 INFO:teuthology.run_tasks:Running task parallel... 2024-03-30T14:23:14.901 INFO:teuthology.task.parallel:starting parallel... 2024-03-30T14:23:14.901 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-30T14:23:14.902 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-30T14:23:14.902 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:23:14.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-30T14:23:14.903 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-30T14:23:14.903 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-30T14:23:14.904 INFO:teuthology.task.exec:Executing custom commands... 2024-03-30T14:23:14.904 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:23:14.904 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-03-30T14:23:14.970 INFO:teuthology.orchestra.run.smithi031.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-30T14:23:14.970 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:23:14.970 INFO:teuthology.orchestra.run.smithi031.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-30T14:23:14.978 INFO:teuthology.orchestra.run.smithi031.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-30T14:23:15.078 INFO:teuthology.orchestra.run.smithi031.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-30T14:23:15.079 INFO:teuthology.orchestra.run.smithi031.stdout:releasing clients 2024-03-30T14:23:15.406 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:15 smithi031 ceph-mon[24911]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T14:23:15.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:15 smithi193 ceph-mon[26933]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T14:23:16.078 INFO:teuthology.orchestra.run.smithi031.stdout: 5 83 18.61 MB/sec warmup 1 sec latency 62.098 ms 2024-03-30T14:23:16.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:16 smithi031 ceph-mon[24911]: pgmap v239: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-30T14:23:16.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:16 smithi193 ceph-mon[26933]: pgmap v239: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-30T14:23:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 179 18.05 MB/sec warmup 2 sec latency 39.364 ms 2024-03-30T14:23:18.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 285 17.62 MB/sec warmup 3 sec latency 44.860 ms 2024-03-30T14:23:18.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-30T14:23:18.374 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:18 smithi031 ceph-mon[24911]: pgmap v240: 97 pgs: 97 active+clean; 8.4 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1014 KiB/s wr, 21 op/s 2024-03-30T14:23:18.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:18 smithi193 ceph-mon[26933]: pgmap v240: 97 pgs: 97 active+clean; 8.4 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1014 KiB/s wr, 21 op/s 2024-03-30T14:23:19.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 380 16.88 MB/sec warmup 4 sec latency 86.725 ms 2024-03-30T14:23:20.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 462 16.18 MB/sec warmup 5 sec latency 70.905 ms 2024-03-30T14:23:20.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:20 smithi031 ceph-mon[24911]: pgmap v241: 97 pgs: 97 active+clean; 16 MiB data, 64 MiB used, 715 GiB / 715 GiB avail; 2.1 MiB/s wr, 190 op/s 2024-03-30T14:23:20.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:20 smithi193 ceph-mon[26933]: pgmap v241: 97 pgs: 97 active+clean; 16 MiB data, 64 MiB used, 715 GiB / 715 GiB avail; 2.1 MiB/s wr, 190 op/s 2024-03-30T14:23:21.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 555 15.91 MB/sec warmup 6 sec latency 86.520 ms 2024-03-30T14:23:21.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-30T14:23:22.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 802 15.31 MB/sec warmup 7 sec latency 23.924 ms 2024-03-30T14:23:22.604 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:22 smithi031 ceph-mon[24911]: pgmap v242: 97 pgs: 97 active+clean; 51 MiB data, 107 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 6.5 MiB/s wr, 284 op/s 2024-03-30T14:23:22.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:22 smithi193 ceph-mon[26933]: pgmap v242: 97 pgs: 97 active+clean; 51 MiB data, 107 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 6.5 MiB/s wr, 284 op/s 2024-03-30T14:23:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 1092 14.22 MB/sec warmup 8 sec latency 86.379 ms 2024-03-30T14:23:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 1778 14.61 MB/sec warmup 9 sec latency 44.487 ms 2024-03-30T14:23:24.432 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:24 smithi031 ceph-mon[24911]: pgmap v243: 97 pgs: 97 active+clean; 93 MiB data, 125 MiB used, 715 GiB / 715 GiB avail; 217 B/s rd, 9.9 MiB/s wr, 329 op/s 2024-03-30T14:23:24.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:24 smithi193 ceph-mon[26933]: pgmap v243: 97 pgs: 97 active+clean; 93 MiB data, 125 MiB used, 715 GiB / 715 GiB avail; 217 B/s rd, 9.9 MiB/s wr, 329 op/s 2024-03-30T14:23:25.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 2460 14.22 MB/sec warmup 10 sec latency 67.541 ms 2024-03-30T14:23:26.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 3246 15.83 MB/sec warmup 11 sec latency 59.103 ms 2024-03-30T14:23:26.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph mgr module enable nfs --force' 2024-03-30T14:23:26.490 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:26 smithi031 ceph-mon[24911]: pgmap v244: 97 pgs: 97 active+clean; 107 MiB data, 157 MiB used, 715 GiB / 715 GiB avail; 196 B/s rd, 11 MiB/s wr, 484 op/s 2024-03-30T14:23:26.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:26 smithi193 ceph-mon[26933]: pgmap v244: 97 pgs: 97 active+clean; 107 MiB data, 157 MiB used, 715 GiB / 715 GiB avail; 196 B/s rd, 11 MiB/s wr, 484 op/s 2024-03-30T14:23:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 3790 16.56 MB/sec warmup 12 sec latency 28.201 ms 2024-03-30T14:23:28.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 4244 15.49 MB/sec warmup 13 sec latency 34.662 ms 2024-03-30T14:23:28.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:28 smithi031 ceph-mon[24911]: pgmap v245: 97 pgs: 97 active+clean; 123 MiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 12 MiB/s wr, 649 op/s 2024-03-30T14:23:28.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:28 smithi193 ceph-mon[26933]: pgmap v245: 97 pgs: 97 active+clean; 123 MiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 12 MiB/s wr, 649 op/s 2024-03-30T14:23:29.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 4871 15.49 MB/sec warmup 14 sec latency 32.397 ms 2024-03-30T14:23:29.418 INFO:teuthology.orchestra.run.smithi031.stderr:module 'nfs' is already enabled 2024-03-30T14:23:29.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:29 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2498140135' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-30T14:23:29.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:29 smithi031 ceph-mon[24911]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-30T14:23:29.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:29 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2498140135' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-30T14:23:29.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:29 smithi193 ceph-mon[26933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-30T14:23:30.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 5702 15.30 MB/sec warmup 15 sec latency 17.462 ms 2024-03-30T14:23:30.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:30 smithi031 ceph-mon[24911]: pgmap v246: 97 pgs: 97 active+clean; 127 MiB data, 208 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 12 MiB/s wr, 700 op/s 2024-03-30T14:23:30.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:30 smithi031 ceph-mon[24911]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-30T14:23:30.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:30 smithi031 ceph-mon[24911]: mgrmap e21: smithi031.lkrchi(active, since 6m), standbys: smithi193.ikzlnb 2024-03-30T14:23:30.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-30T14:23:30.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:30 smithi193 ceph-mon[26933]: pgmap v246: 97 pgs: 97 active+clean; 127 MiB data, 208 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 12 MiB/s wr, 700 op/s 2024-03-30T14:23:30.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:30 smithi193 ceph-mon[26933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-30T14:23:30.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:30 smithi193 ceph-mon[26933]: mgrmap e21: smithi031.lkrchi(active, since 6m), standbys: smithi193.ikzlnb 2024-03-30T14:23:31.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 6252 15.43 MB/sec warmup 16 sec latency 77.271 ms 2024-03-30T14:23:32.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 6856 15.97 MB/sec warmup 17 sec latency 49.116 ms 2024-03-30T14:23:32.554 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:32 smithi031 ceph-mon[24911]: pgmap v247: 97 pgs: 97 active+clean; 130 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 12 MiB/s wr, 668 op/s 2024-03-30T14:23:32.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:32 smithi193 ceph-mon[26933]: pgmap v247: 97 pgs: 97 active+clean; 130 MiB data, 211 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 12 MiB/s wr, 668 op/s 2024-03-30T14:23:33.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 7327 16.35 MB/sec warmup 18 sec latency 40.583 ms 2024-03-30T14:23:33.912 INFO:teuthology.orchestra.run.smithi031.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:23:34.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 7612 15.58 MB/sec warmup 19 sec latency 33.098 ms 2024-03-30T14:23:34.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:34 smithi193 ceph-mon[26933]: pgmap v248: 97 pgs: 97 active+clean; 131 MiB data, 260 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 11 MiB/s wr, 709 op/s 2024-03-30T14:23:34.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:34 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:34.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:34 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:34.577 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:34 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:34.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:34 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:34.618 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:34 smithi031 ceph-mon[24911]: pgmap v248: 97 pgs: 97 active+clean; 131 MiB data, 260 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 11 MiB/s wr, 709 op/s 2024-03-30T14:23:34.618 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:34 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:34.619 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:34 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:34.619 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:34 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:34.619 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:34 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 8039 14.99 MB/sec warmup 20 sec latency 34.309 ms 2024-03-30T14:23:35.175 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-30T14:23:35.176 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:23:35.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-30T14:23:35.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:35 smithi031 ceph-mon[24911]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:35.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:35 smithi031 ceph-mon[24911]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:23:35.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:35 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:35.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:35 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:35.632 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:35 smithi031 ceph-mon[24911]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:23:35.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:35 smithi193 ceph-mon[26933]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:35.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:35 smithi193 ceph-mon[26933]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:23:35.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:35 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:35.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:35 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:23:35.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:35 smithi193 ceph-mon[26933]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:23:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 8420 14.88 MB/sec warmup 21 sec latency 92.002 ms 2024-03-30T14:23:36.661 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:36 smithi031 ceph-mon[24911]: pgmap v249: 97 pgs: 97 active+clean; 137 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 8.6 MiB/s wr, 770 op/s 2024-03-30T14:23:36.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:36 smithi193 ceph-mon[26933]: pgmap v249: 97 pgs: 97 active+clean; 137 MiB data, 301 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 8.6 MiB/s wr, 770 op/s 2024-03-30T14:23:37.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 8992 14.69 MB/sec warmup 22 sec latency 94.127 ms 2024-03-30T14:23:38.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 9493 14.39 MB/sec warmup 23 sec latency 26.528 ms 2024-03-30T14:23:38.563 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:38 smithi031 ceph-mon[24911]: pgmap v250: 97 pgs: 97 active+clean; 150 MiB data, 360 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 836 op/s 2024-03-30T14:23:38.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:38 smithi193 ceph-mon[26933]: pgmap v250: 97 pgs: 97 active+clean; 150 MiB data, 360 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 836 op/s 2024-03-30T14:23:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:23:39.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 9912 14.65 MB/sec warmup 24 sec latency 24.973 ms 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (5m) 27s ago 6m 24.8M - ba2b418f427c 9fc4502dff1a 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (6m) 27s ago 6m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (6m) 30s ago 6m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (5m) 27s ago 6m 53.9M - 8.3.5 dad864ee21e9 c132179ebde6 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (60s) 27s ago 59s 15.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (57s) 30s ago 57s 12.1M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:9283 running (8m) 27s ago 8m 460M - 17.2.0 e1d6a67b021e 330ec216c990 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (6m) 30s ago 6m 413M - 17.2.0 e1d6a67b021e def8a6a77a9b 2024-03-30T14:23:40.070 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (8m) 27s ago 8m 44.1M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (6m) 30s ago 6m 37.9M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (38s) 27s ago 38s 50.9M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (33s) 30s ago 32s 46.7M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (6m) 27s ago 6m 17.7M - 1dbe0e931976 a5b6d8e0efa4 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (6m) 30s ago 5m 17.4M - 1dbe0e931976 3f7d1e5ad9be 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (5m) 27s ago 5m 42.1M 3217M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (4m) 27s ago 4m 43.1M 3217M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (4m) 27s ago 4m 42.2M 3217M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (4m) 27s ago 4m 40.7M 3217M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (3m) 30s ago 3m 42.5M 3984M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (3m) 30s ago 3m 39.4M 3984M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:23:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (2m) 30s ago 2m 40.8M 3984M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:23:40.072 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (2m) 30s ago 2m 40.9M 3984M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:23:40.072 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (5m) 27s ago 5m 62.1M - 514e6a882f6e 15161d5c677a 2024-03-30T14:23:40.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 10433 14.72 MB/sec warmup 25 sec latency 24.983 ms 2024-03-30T14:23:40.572 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:40 smithi031 ceph-mon[24911]: pgmap v251: 97 pgs: 97 active+clean; 149 MiB data, 365 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.7 MiB/s wr, 612 op/s 2024-03-30T14:23:40.572 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:40 smithi031 ceph-mon[24911]: from='client.24387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:40.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:40 smithi031 ceph-mon[24911]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:23:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:23:40.742 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:23:40.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:40 smithi193 ceph-mon[26933]: pgmap v251: 97 pgs: 97 active+clean; 149 MiB data, 365 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.7 MiB/s wr, 612 op/s 2024-03-30T14:23:40.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:40 smithi193 ceph-mon[26933]: from='client.24387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:40.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:40 smithi193 ceph-mon[26933]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:41.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 10867 15.00 MB/sec warmup 26 sec latency 32.828 ms 2024-03-30T14:23:41.296 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:23:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb", 2024-03-30T14:23:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:23:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:23:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "", 2024-03-30T14:23:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb image" 2024-03-30T14:23:41.297 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:23:41.595 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:41 smithi031 ceph-mon[24911]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:41.595 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:41 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1971392324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:23:41.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:41 smithi193 ceph-mon[26933]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:41.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:41 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1971392324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:23:41.937 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:23:42.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 11119 14.50 MB/sec warmup 27 sec latency 32.863 ms 2024-03-30T14:23:42.598 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:42 smithi031 ceph-mon[24911]: pgmap v252: 97 pgs: 97 active+clean; 146 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 MiB/s wr, 592 op/s 2024-03-30T14:23:42.598 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:42 smithi031 ceph-mon[24911]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:42.599 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:42 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1088825817' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:23:42.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:42 smithi193 ceph-mon[26933]: pgmap v252: 97 pgs: 97 active+clean; 146 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 MiB/s wr, 592 op/s 2024-03-30T14:23:42.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:42 smithi193 ceph-mon[26933]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:23:42.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:42 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1088825817' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:23:43.079 INFO:teuthology.orchestra.run.smithi031.stdout: 5 11550 14.12 MB/sec warmup 28 sec latency 28.173 ms 2024-03-30T14:23:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 11925 13.97 MB/sec warmup 29 sec latency 29.411 ms 2024-03-30T14:23:44.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:44 smithi031 ceph-mon[24911]: pgmap v253: 97 pgs: 97 active+clean; 151 MiB data, 423 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.7 MiB/s wr, 726 op/s 2024-03-30T14:23:44.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:44 smithi193 ceph-mon[26933]: pgmap v253: 97 pgs: 97 active+clean; 151 MiB data, 423 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.7 MiB/s wr, 726 op/s 2024-03-30T14:23:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 12387 13.94 MB/sec warmup 30 sec latency 43.169 ms 2024-03-30T14:23:46.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 12964 13.69 MB/sec warmup 31 sec latency 29.760 ms 2024-03-30T14:23:46.685 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:46 smithi031 ceph-mon[24911]: pgmap v254: 97 pgs: 97 active+clean; 156 MiB data, 431 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.9 MiB/s wr, 656 op/s 2024-03-30T14:23:46.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:46 smithi193 ceph-mon[26933]: pgmap v254: 97 pgs: 97 active+clean; 156 MiB data, 431 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.9 MiB/s wr, 656 op/s 2024-03-30T14:23:47.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 13299 13.62 MB/sec warmup 32 sec latency 34.938 ms 2024-03-30T14:23:48.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 14014 14.14 MB/sec warmup 33 sec latency 25.675 ms 2024-03-30T14:23:48.707 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:48 smithi031 ceph-mon[24911]: pgmap v255: 97 pgs: 97 active+clean; 159 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-03-30T14:23:48.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:48 smithi193 ceph-mon[26933]: pgmap v255: 97 pgs: 97 active+clean; 159 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-03-30T14:23:49.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 14454 14.29 MB/sec warmup 34 sec latency 33.278 ms 2024-03-30T14:23:50.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 14806 13.94 MB/sec warmup 35 sec latency 31.062 ms 2024-03-30T14:23:50.749 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:50 smithi031 ceph-mon[24911]: pgmap v256: 97 pgs: 97 active+clean; 159 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.2 MiB/s wr, 516 op/s 2024-03-30T14:23:50.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:50 smithi193 ceph-mon[26933]: pgmap v256: 97 pgs: 97 active+clean; 159 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.2 MiB/s wr, 516 op/s 2024-03-30T14:23:51.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 15176 13.68 MB/sec warmup 36 sec latency 78.485 ms 2024-03-30T14:23:52.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 15584 13.70 MB/sec warmup 37 sec latency 112.629 ms 2024-03-30T14:23:52.523 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:52 smithi031 ceph-mon[24911]: pgmap v257: 97 pgs: 97 active+clean; 159 MiB data, 523 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.3 MiB/s wr, 588 op/s 2024-03-30T14:23:52.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:52 smithi193 ceph-mon[26933]: pgmap v257: 97 pgs: 97 active+clean; 159 MiB data, 523 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.3 MiB/s wr, 588 op/s 2024-03-30T14:23:53.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 16127 13.55 MB/sec warmup 38 sec latency 30.197 ms 2024-03-30T14:23:54.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 16590 13.41 MB/sec warmup 39 sec latency 48.392 ms 2024-03-30T14:23:54.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:54 smithi031 ceph-mon[24911]: pgmap v258: 97 pgs: 97 active+clean; 170 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 699 op/s 2024-03-30T14:23:54.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:54 smithi193 ceph-mon[26933]: pgmap v258: 97 pgs: 97 active+clean; 170 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 699 op/s 2024-03-30T14:23:55.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 17201 13.77 MB/sec warmup 40 sec latency 43.809 ms 2024-03-30T14:23:56.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 17847 14.06 MB/sec warmup 41 sec latency 21.852 ms 2024-03-30T14:23:56.560 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:56 smithi031 ceph-mon[24911]: pgmap v259: 97 pgs: 97 active+clean; 167 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.0 MiB/s wr, 516 op/s 2024-03-30T14:23:56.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:56 smithi193 ceph-mon[26933]: pgmap v259: 97 pgs: 97 active+clean; 167 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.0 MiB/s wr, 516 op/s 2024-03-30T14:23:57.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 18201 13.88 MB/sec warmup 42 sec latency 33.236 ms 2024-03-30T14:23:58.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 18578 13.63 MB/sec warmup 43 sec latency 30.819 ms 2024-03-30T14:23:58.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:23:58 smithi031 ceph-mon[24911]: pgmap v260: 97 pgs: 97 active+clean; 171 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 688 op/s 2024-03-30T14:23:58.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:23:58 smithi193 ceph-mon[26933]: pgmap v260: 97 pgs: 97 active+clean; 171 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 688 op/s 2024-03-30T14:23:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 18942 13.54 MB/sec warmup 44 sec latency 28.604 ms 2024-03-30T14:24:00.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 19484 13.54 MB/sec warmup 45 sec latency 47.936 ms 2024-03-30T14:24:00.615 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:00 smithi031 ceph-mon[24911]: pgmap v261: 97 pgs: 97 active+clean; 172 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.8 MiB/s wr, 483 op/s 2024-03-30T14:24:00.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:00 smithi193 ceph-mon[26933]: pgmap v261: 97 pgs: 97 active+clean; 172 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.8 MiB/s wr, 483 op/s 2024-03-30T14:24:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 20076 13.37 MB/sec warmup 46 sec latency 41.495 ms 2024-03-30T14:24:02.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 20626 13.59 MB/sec warmup 47 sec latency 30.618 ms 2024-03-30T14:24:02.567 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:02 smithi031 ceph-mon[24911]: pgmap v262: 97 pgs: 97 active+clean; 186 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 7.7 MiB/s wr, 738 op/s 2024-03-30T14:24:02.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:02 smithi193 ceph-mon[26933]: pgmap v262: 97 pgs: 97 active+clean; 186 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 7.7 MiB/s wr, 738 op/s 2024-03-30T14:24:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 21215 13.79 MB/sec warmup 48 sec latency 23.871 ms 2024-03-30T14:24:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 21616 13.82 MB/sec warmup 49 sec latency 43.630 ms 2024-03-30T14:24:04.786 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:04 smithi031 ceph-mon[24911]: pgmap v263: 97 pgs: 97 active+clean; 181 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 752 op/s 2024-03-30T14:24:04.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:04 smithi193 ceph-mon[26933]: pgmap v263: 97 pgs: 97 active+clean; 181 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 752 op/s 2024-03-30T14:24:05.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 21903 13.59 MB/sec warmup 50 sec latency 39.763 ms 2024-03-30T14:24:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 22327 13.43 MB/sec warmup 51 sec latency 47.421 ms 2024-03-30T14:24:06.582 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:06 smithi031 ceph-mon[24911]: pgmap v264: 97 pgs: 97 active+clean; 181 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.4 MiB/s wr, 631 op/s 2024-03-30T14:24:06.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:06 smithi193 ceph-mon[26933]: pgmap v264: 97 pgs: 97 active+clean; 181 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.4 MiB/s wr, 631 op/s 2024-03-30T14:24:07.083 INFO:teuthology.orchestra.run.smithi031.stdout: 5 22689 13.37 MB/sec warmup 52 sec latency 72.351 ms 2024-03-30T14:24:08.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 23304 13.35 MB/sec warmup 53 sec latency 46.873 ms 2024-03-30T14:24:08.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:08 smithi031 ceph-mon[24911]: pgmap v265: 97 pgs: 97 active+clean; 197 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 853 op/s 2024-03-30T14:24:08.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:08 smithi193 ceph-mon[26933]: pgmap v265: 97 pgs: 97 active+clean; 197 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 853 op/s 2024-03-30T14:24:09.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 23759 13.24 MB/sec warmup 54 sec latency 89.620 ms 2024-03-30T14:24:10.080 INFO:teuthology.orchestra.run.smithi031.stdout: 5 24291 13.45 MB/sec warmup 55 sec latency 34.458 ms 2024-03-30T14:24:10.621 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:10 smithi031 ceph-mon[24911]: pgmap v266: 97 pgs: 97 active+clean; 197 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.3 MiB/s wr, 654 op/s 2024-03-30T14:24:10.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:10 smithi193 ceph-mon[26933]: pgmap v266: 97 pgs: 97 active+clean; 197 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.3 MiB/s wr, 654 op/s 2024-03-30T14:24:11.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 24837 13.55 MB/sec warmup 56 sec latency 39.797 ms 2024-03-30T14:24:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 25344 13.58 MB/sec warmup 57 sec latency 29.357 ms 2024-03-30T14:24:12.447 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:24:12.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:12 smithi031 ceph-mon[24911]: pgmap v267: 97 pgs: 97 active+clean; 195 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-03-30T14:24:12.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:12 smithi193 ceph-mon[26933]: pgmap v267: 97 pgs: 97 active+clean; 195 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-03-30T14:24:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 25904 13.46 MB/sec warmup 58 sec latency 71.628 ms 2024-03-30T14:24:13.295 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:24:13.295 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (5m) 60s ago 7m 24.8M - ba2b418f427c 9fc4502dff1a 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (7m) 60s ago 7m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (6m) 63s ago 6m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (5m) 60s ago 7m 53.9M - 8.3.5 dad864ee21e9 c132179ebde6 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (93s) 60s ago 92s 15.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (90s) 63s ago 90s 12.1M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:9283 running (8m) 60s ago 8m 460M - 17.2.0 e1d6a67b021e 330ec216c990 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (6m) 63s ago 6m 413M - 17.2.0 e1d6a67b021e def8a6a77a9b 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (8m) 60s ago 8m 44.1M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (6m) 63s ago 6m 37.9M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (71s) 60s ago 71s 50.9M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (66s) 63s ago 66s 46.7M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (7m) 60s ago 7m 17.7M - 1dbe0e931976 a5b6d8e0efa4 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (6m) 63s ago 6m 17.4M - 1dbe0e931976 3f7d1e5ad9be 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (6m) 60s ago 6m 42.1M 3217M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:24:13.296 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (5m) 60s ago 5m 43.1M 3217M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:24:13.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (5m) 60s ago 5m 42.2M 3217M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:24:13.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (4m) 60s ago 4m 40.7M 3217M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:24:13.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (4m) 63s ago 4m 42.5M 3984M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:24:13.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (3m) 63s ago 3m 39.4M 3984M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:24:13.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (3m) 63s ago 3m 40.8M 3984M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:24:13.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (3m) 63s ago 3m 40.9M 3984M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:24:13.297 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (6m) 60s ago 6m 62.1M - 514e6a882f6e 15161d5c677a 2024-03-30T14:24:13.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:13 smithi031 ceph-mon[24911]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:13.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:24:13.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:24:13.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:13.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:24:13.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:13.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:13 smithi193 ceph-mon[26933]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:13.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:24:13.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:24:13.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:13.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:24:13.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:13.842 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:24:13.842 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-30T14:24:13.843 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:24:13.844 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:24:14.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 26508 13.57 MB/sec warmup 59 sec latency 44.655 ms 2024-03-30T14:24:14.340 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:24:14.341 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:24:14.341 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:24:14.341 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:24:14.341 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "0/23 daemons upgraded", 2024-03-30T14:24:14.341 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d image on host smithi193" 2024-03-30T14:24:14.341 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:24:14.710 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: pgmap v268: 97 pgs: 97 active+clean; 200 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 648 op/s 2024-03-30T14:24:14.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:14.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: Upgrade: Target is version 19.0.0-1967-g4e50bbcd (unknown) 2024-03-30T14:24:14.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d'] 2024-03-30T14:24:14.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: Upgrade: Need to upgrade myself (mgr.smithi031.lkrchi) 2024-03-30T14:24:14.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:14.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d on smithi193 2024-03-30T14:24:14.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:14 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/4038570978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:24:14.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: pgmap v268: 97 pgs: 97 active+clean; 200 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 648 op/s 2024-03-30T14:24:14.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:14.827 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: Upgrade: Target is version 19.0.0-1967-g4e50bbcd (unknown) 2024-03-30T14:24:14.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d'] 2024-03-30T14:24:14.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: Upgrade: Need to upgrade myself (mgr.smithi031.lkrchi) 2024-03-30T14:24:14.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:14.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d on smithi193 2024-03-30T14:24:14.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:14 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/4038570978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:24:14.928 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:24:15.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 27302 13.53 MB/sec warmup 60 sec latency 20.644 ms 2024-03-30T14:24:15.729 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:15 smithi031 ceph-mon[24911]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:15.730 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:15 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2885809829' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:24:16.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:15 smithi193 ceph-mon[26933]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:16.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:15 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2885809829' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:24:16.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 28011 13.81 MB/sec warmup 61 sec latency 38.703 ms 2024-03-30T14:24:16.738 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:16 smithi031 ceph-mon[24911]: pgmap v269: 97 pgs: 97 active+clean; 209 MiB data, 833 MiB used, 714 GiB / 715 GiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 728 op/s 2024-03-30T14:24:17.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:16 smithi193 ceph-mon[26933]: pgmap v269: 97 pgs: 97 active+clean; 209 MiB data, 833 MiB used, 714 GiB / 715 GiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 728 op/s 2024-03-30T14:24:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 28594 13.95 MB/sec warmup 62 sec latency 29.287 ms 2024-03-30T14:24:18.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 28926 13.81 MB/sec warmup 63 sec latency 42.956 ms 2024-03-30T14:24:18.778 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:18 smithi031 ceph-mon[24911]: pgmap v270: 97 pgs: 97 active+clean; 213 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 884 op/s 2024-03-30T14:24:19.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:18 smithi193 ceph-mon[26933]: pgmap v270: 97 pgs: 97 active+clean; 213 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 884 op/s 2024-03-30T14:24:19.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 29384 13.72 MB/sec warmup 64 sec latency 26.546 ms 2024-03-30T14:24:20.081 INFO:teuthology.orchestra.run.smithi031.stdout: 5 29963 13.65 MB/sec warmup 65 sec latency 21.183 ms 2024-03-30T14:24:20.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:20 smithi031 ceph-mon[24911]: pgmap v271: 97 pgs: 97 active+clean; 218 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 732 op/s 2024-03-30T14:24:21.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:20 smithi193 ceph-mon[26933]: pgmap v271: 97 pgs: 97 active+clean; 218 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 732 op/s 2024-03-30T14:24:21.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 30618 13.66 MB/sec warmup 66 sec latency 31.976 ms 2024-03-30T14:24:22.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 31177 13.72 MB/sec warmup 67 sec latency 34.778 ms 2024-03-30T14:24:22.828 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:22 smithi031 ceph-mon[24911]: pgmap v272: 97 pgs: 97 active+clean; 223 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.4 MiB/s wr, 836 op/s 2024-03-30T14:24:23.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:22 smithi193 ceph-mon[26933]: pgmap v272: 97 pgs: 97 active+clean; 223 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.4 MiB/s wr, 836 op/s 2024-03-30T14:24:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 32059 14.07 MB/sec warmup 68 sec latency 21.933 ms 2024-03-30T14:24:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 32795 14.04 MB/sec warmup 69 sec latency 15.894 ms 2024-03-30T14:24:24.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:24 smithi031 ceph-mon[24911]: pgmap v273: 97 pgs: 97 active+clean; 219 MiB data, 935 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 851 op/s 2024-03-30T14:24:25.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:24 smithi193 ceph-mon[26933]: pgmap v273: 97 pgs: 97 active+clean; 219 MiB data, 935 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 851 op/s 2024-03-30T14:24:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 33532 14.08 MB/sec warmup 70 sec latency 27.281 ms 2024-03-30T14:24:26.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 34181 14.05 MB/sec warmup 71 sec latency 36.457 ms 2024-03-30T14:24:26.804 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:26 smithi031 ceph-mon[24911]: pgmap v274: 97 pgs: 97 active+clean; 220 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 705 op/s 2024-03-30T14:24:27.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:26 smithi193 ceph-mon[26933]: pgmap v274: 97 pgs: 97 active+clean; 220 MiB data, 983 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 705 op/s 2024-03-30T14:24:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 34794 14.12 MB/sec warmup 72 sec latency 31.147 ms 2024-03-30T14:24:28.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 35663 14.46 MB/sec warmup 73 sec latency 18.431 ms 2024-03-30T14:24:28.814 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:28 smithi031 ceph-mon[24911]: pgmap v275: 97 pgs: 97 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 811 op/s 2024-03-30T14:24:29.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:28 smithi193 ceph-mon[26933]: pgmap v275: 97 pgs: 97 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 811 op/s 2024-03-30T14:24:29.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 36256 14.38 MB/sec warmup 74 sec latency 18.736 ms 2024-03-30T14:24:30.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 36895 14.37 MB/sec warmup 75 sec latency 31.403 ms 2024-03-30T14:24:30.698 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:30 smithi031 ceph-mon[24911]: pgmap v276: 97 pgs: 97 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 815 op/s 2024-03-30T14:24:31.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:30 smithi193 ceph-mon[26933]: pgmap v276: 97 pgs: 97 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 815 op/s 2024-03-30T14:24:31.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 37744 14.40 MB/sec warmup 76 sec latency 21.605 ms 2024-03-30T14:24:32.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 38577 14.60 MB/sec warmup 77 sec latency 24.913 ms 2024-03-30T14:24:32.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:32 smithi031 ceph-mon[24911]: pgmap v277: 97 pgs: 97 active+clean; 235 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 770 op/s 2024-03-30T14:24:33.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:32 smithi193 ceph-mon[26933]: pgmap v277: 97 pgs: 97 active+clean; 235 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 770 op/s 2024-03-30T14:24:33.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 39306 14.78 MB/sec warmup 78 sec latency 15.507 ms 2024-03-30T14:24:34.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 39950 14.70 MB/sec warmup 79 sec latency 21.254 ms 2024-03-30T14:24:34.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:34 smithi031 ceph-mon[24911]: pgmap v278: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 845 op/s 2024-03-30T14:24:34.912 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:34 smithi193 ceph-mon[26933]: pgmap v278: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 845 op/s 2024-03-30T14:24:35.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 40804 14.80 MB/sec warmup 80 sec latency 14.484 ms 2024-03-30T14:24:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 41585 14.80 MB/sec warmup 81 sec latency 28.161 ms 2024-03-30T14:24:36.627 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:36 smithi031 ceph-mon[24911]: pgmap v279: 97 pgs: 97 active+clean; 249 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 980 op/s 2024-03-30T14:24:37.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:36 smithi193 ceph-mon[26933]: pgmap v279: 97 pgs: 97 active+clean; 249 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 980 op/s 2024-03-30T14:24:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: 5 42026 14.79 MB/sec warmup 82 sec latency 31.093 ms 2024-03-30T14:24:38.091 INFO:teuthology.orchestra.run.smithi031.stdout: 5 42569 14.91 MB/sec warmup 83 sec latency 46.738 ms 2024-03-30T14:24:38.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:38 smithi031 ceph-mon[24911]: pgmap v280: 97 pgs: 97 active+clean; 251 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-03-30T14:24:39.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:38 smithi193 ceph-mon[26933]: pgmap v280: 97 pgs: 97 active+clean; 251 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-03-30T14:24:39.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 43038 14.90 MB/sec warmup 84 sec latency 24.886 ms 2024-03-30T14:24:40.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 43474 14.78 MB/sec warmup 85 sec latency 22.400 ms 2024-03-30T14:24:40.756 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:40 smithi031 ceph-mon[24911]: pgmap v281: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 909 op/s 2024-03-30T14:24:41.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:40 smithi193 ceph-mon[26933]: pgmap v281: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 909 op/s 2024-03-30T14:24:41.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 44038 14.76 MB/sec warmup 86 sec latency 33.739 ms 2024-03-30T14:24:42.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 44671 14.75 MB/sec warmup 87 sec latency 31.445 ms 2024-03-30T14:24:42.773 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:42 smithi031 ceph-mon[24911]: pgmap v282: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 796 op/s 2024-03-30T14:24:43.077 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:42 smithi193 ceph-mon[26933]: pgmap v282: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 796 op/s 2024-03-30T14:24:43.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 45392 14.80 MB/sec warmup 88 sec latency 33.954 ms 2024-03-30T14:24:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 46056 14.95 MB/sec warmup 89 sec latency 43.454 ms 2024-03-30T14:24:44.839 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:44 smithi031 ceph-mon[24911]: pgmap v283: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 950 op/s 2024-03-30T14:24:45.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:44 smithi193 ceph-mon[26933]: pgmap v283: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 950 op/s 2024-03-30T14:24:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: 5 46610 14.97 MB/sec warmup 90 sec latency 40.087 ms 2024-03-30T14:24:45.412 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:24:46.091 INFO:teuthology.orchestra.run.smithi031.stdout: 5 46928 14.86 MB/sec warmup 91 sec latency 105.235 ms 2024-03-30T14:24:46.252 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:24:46.252 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (6m) 93s ago 8m 24.8M - ba2b418f427c 9fc4502dff1a 2024-03-30T14:24:46.252 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (7m) 93s ago 7m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:24:46.252 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (7m) 96s ago 7m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (6m) 93s ago 7m 53.9M - 8.3.5 dad864ee21e9 c132179ebde6 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (2m) 93s ago 2m 15.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (2m) 96s ago 2m 12.1M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:9283 running (9m) 93s ago 9m 460M - 17.2.0 e1d6a67b021e 330ec216c990 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (7m) 96s ago 7m 413M - 17.2.0 e1d6a67b021e def8a6a77a9b 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (9m) 93s ago 9m 44.1M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (7m) 96s ago 7m 37.9M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (104s) 93s ago 104s 50.9M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (99s) 96s ago 98s 46.7M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (7m) 93s ago 7m 17.7M - 1dbe0e931976 a5b6d8e0efa4 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (7m) 96s ago 7m 17.4M - 1dbe0e931976 3f7d1e5ad9be 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (6m) 93s ago 6m 42.1M 3217M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (6m) 93s ago 6m 43.1M 3217M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (5m) 93s ago 5m 42.2M 3217M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (5m) 93s ago 5m 40.7M 3217M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:24:46.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (4m) 96s ago 4m 42.5M 3984M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:24:46.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (4m) 96s ago 4m 39.4M 3984M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:24:46.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (4m) 96s ago 3m 40.8M 3984M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:24:46.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (3m) 96s ago 3m 40.9M 3984M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:24:46.254 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (6m) 93s ago 6m 62.1M - 514e6a882f6e 15161d5c677a 2024-03-30T14:24:46.811 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:24:46.811 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:24:46.811 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:24:46.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:46.811 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:24:46.812 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:24:46.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:46 smithi031 ceph-mon[24911]: pgmap v284: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 860 op/s 2024-03-30T14:24:46.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:46 smithi031 ceph-mon[24911]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:46.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:46 smithi193 ceph-mon[26933]: pgmap v284: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 860 op/s 2024-03-30T14:24:46.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:46 smithi193 ceph-mon[26933]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:47.091 INFO:teuthology.orchestra.run.smithi031.stdout: 5 47528 14.81 MB/sec warmup 92 sec latency 28.963 ms 2024-03-30T14:24:47.291 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:24:47.291 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:24:47.291 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:24:47.292 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:24:47.292 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "0/23 daemons upgraded", 2024-03-30T14:24:47.292 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d image on host smithi193" 2024-03-30T14:24:47.292 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:24:47.822 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:24:47.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:47 smithi031 ceph-mon[24911]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:47.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:47 smithi031 ceph-mon[24911]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:47.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:47 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/802252118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:24:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:47 smithi193 ceph-mon[26933]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:47 smithi193 ceph-mon[26933]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:48.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:47 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/802252118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:24:48.091 INFO:teuthology.orchestra.run.smithi031.stdout: 5 48111 14.82 MB/sec warmup 93 sec latency 23.659 ms 2024-03-30T14:24:48.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:48 smithi031 ceph-mon[24911]: pgmap v285: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 901 op/s 2024-03-30T14:24:48.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:48 smithi031 ceph-mon[24911]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:48.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:48 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/305026962' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:24:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:48 smithi193 ceph-mon[26933]: pgmap v285: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 901 op/s 2024-03-30T14:24:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:48 smithi193 ceph-mon[26933]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:24:49.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:48 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/305026962' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:24:49.091 INFO:teuthology.orchestra.run.smithi031.stdout: 5 48766 14.82 MB/sec warmup 94 sec latency 60.113 ms 2024-03-30T14:24:50.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 49387 14.96 MB/sec warmup 95 sec latency 43.684 ms 2024-03-30T14:24:50.905 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:50 smithi031 ceph-mon[24911]: pgmap v286: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.5 MiB/s wr, 739 op/s 2024-03-30T14:24:50.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:50 smithi193 ceph-mon[26933]: pgmap v286: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.5 MiB/s wr, 739 op/s 2024-03-30T14:24:51.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 50069 15.01 MB/sec warmup 96 sec latency 19.151 ms 2024-03-30T14:24:52.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 50513 14.93 MB/sec warmup 97 sec latency 56.383 ms 2024-03-30T14:24:52.929 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:52 smithi031 ceph-mon[24911]: pgmap v287: 97 pgs: 97 active+clean; 281 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.5 MiB/s wr, 816 op/s 2024-03-30T14:24:53.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:52 smithi193 ceph-mon[26933]: pgmap v287: 97 pgs: 97 active+clean; 281 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.5 MiB/s wr, 816 op/s 2024-03-30T14:24:53.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 51165 14.89 MB/sec warmup 98 sec latency 48.233 ms 2024-03-30T14:24:54.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 51866 14.94 MB/sec warmup 99 sec latency 28.453 ms 2024-03-30T14:24:54.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:54 smithi031 ceph-mon[24911]: pgmap v288: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 931 op/s 2024-03-30T14:24:54.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:54 smithi031 ceph-mon[24911]: Upgrade: Updating mgr.smithi193.ikzlnb 2024-03-30T14:24:54.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:54 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:54.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:54 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:24:54.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:54 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:24:54.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:54 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:24:54.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:54 smithi031 ceph-mon[24911]: Deploying daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:24:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:54 smithi193 ceph-mon[26933]: pgmap v288: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 931 op/s 2024-03-30T14:24:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:54 smithi193 ceph-mon[26933]: Upgrade: Updating mgr.smithi193.ikzlnb 2024-03-30T14:24:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:54 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:54 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:24:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:54 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:24:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:54 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:24:54.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:54 smithi193 ceph-mon[26933]: Deploying daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:24:55.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 52557 14.97 MB/sec warmup 100 sec latency 31.659 ms 2024-03-30T14:24:56.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 53164 15.07 MB/sec warmup 101 sec latency 26.990 ms 2024-03-30T14:24:56.464 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:56 smithi031 ceph-mon[24911]: pgmap v289: 97 pgs: 97 active+clean; 291 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.5 MiB/s wr, 827 op/s 2024-03-30T14:24:56.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:56 smithi193 ceph-mon[26933]: pgmap v289: 97 pgs: 97 active+clean; 291 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.5 MiB/s wr, 827 op/s 2024-03-30T14:24:57.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 53716 15.08 MB/sec warmup 102 sec latency 33.736 ms 2024-03-30T14:24:58.092 INFO:teuthology.orchestra.run.smithi031.stdout: 5 54025 14.98 MB/sec warmup 103 sec latency 33.740 ms 2024-03-30T14:24:58.728 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:58 smithi193 ceph-mon[26933]: pgmap v290: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 876 op/s 2024-03-30T14:24:58.729 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:58 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:58.729 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:58 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:24:58.729 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:58 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:24:58.729 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:24:58 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:24:58.739 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:58 smithi031 ceph-mon[24911]: pgmap v290: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 876 op/s 2024-03-30T14:24:58.739 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:58 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:24:58.739 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:58 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:24:58.739 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:58 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:24:58.740 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:24:58 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:24:59.097 INFO:teuthology.orchestra.run.smithi031.stdout: 5 54583 14.93 MB/sec warmup 104 sec latency 34.187 ms 2024-03-30T14:25:00.097 INFO:teuthology.orchestra.run.smithi031.stdout: 5 55078 14.91 MB/sec warmup 105 sec latency 39.445 ms 2024-03-30T14:25:00.763 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:00 smithi031 ceph-mon[24911]: pgmap v291: 97 pgs: 97 active+clean; 300 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 758 op/s 2024-03-30T14:25:00.802 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:00 smithi193 ceph-mon[26933]: pgmap v291: 97 pgs: 97 active+clean; 300 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 758 op/s 2024-03-30T14:25:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: 5 55702 14.92 MB/sec warmup 106 sec latency 43.731 ms 2024-03-30T14:25:02.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 56241 14.95 MB/sec warmup 107 sec latency 30.078 ms 2024-03-30T14:25:02.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:02 smithi031 ceph-mon[24911]: pgmap v292: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 920 op/s 2024-03-30T14:25:02.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:02 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:02.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:02 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:02.709 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:02 smithi193 ceph-mon[26933]: pgmap v292: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 920 op/s 2024-03-30T14:25:02.710 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:02 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:02.710 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:02 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: 5 56623 14.95 MB/sec warmup 108 sec latency 51.963 ms 2024-03-30T14:25:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 57154 14.97 MB/sec warmup 109 sec latency 23.456 ms 2024-03-30T14:25:04.476 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:04 smithi031 ceph-mon[24911]: pgmap v293: 97 pgs: 97 active+clean; 299 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 801 op/s 2024-03-30T14:25:04.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:04 smithi193 ceph-mon[26933]: pgmap v293: 97 pgs: 97 active+clean; 299 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 801 op/s 2024-03-30T14:25:05.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 57593 14.90 MB/sec warmup 110 sec latency 20.866 ms 2024-03-30T14:25:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 58157 14.85 MB/sec warmup 111 sec latency 36.300 ms 2024-03-30T14:25:06.553 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:06 smithi193 ceph-mon[26933]: pgmap v294: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 651 op/s 2024-03-30T14:25:06.553 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:06 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:25:06.553 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:06 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:25:06.553 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:06 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:25:06.553 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:06 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:25:06.553 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:06 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:25:06.553 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:06 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:25:06.607 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:06 smithi031 ceph-mon[24911]: pgmap v294: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 651 op/s 2024-03-30T14:25:06.608 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:06 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:25:06.608 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:06 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:25:06.608 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:06 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:25:06.608 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:06 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:25:06.608 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:06 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:25:06.608 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:06 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:25:07.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 58630 14.83 MB/sec warmup 112 sec latency 44.563 ms 2024-03-30T14:25:07.615 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:07 smithi031 ceph-mon[24911]: mgrmap e22: smithi031.lkrchi(active, since 7m), standbys: smithi193.ikzlnb 2024-03-30T14:25:07.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:07 smithi193 ceph-mon[26933]: mgrmap e22: smithi031.lkrchi(active, since 7m), standbys: smithi193.ikzlnb 2024-03-30T14:25:08.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 59215 14.84 MB/sec warmup 113 sec latency 43.935 ms 2024-03-30T14:25:08.634 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:08 smithi031 ceph-mon[24911]: pgmap v295: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 735 op/s 2024-03-30T14:25:08.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:08 smithi193 ceph-mon[26933]: pgmap v295: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 735 op/s 2024-03-30T14:25:09.101 INFO:teuthology.orchestra.run.smithi031.stdout: 5 59713 14.87 MB/sec warmup 114 sec latency 48.500 ms 2024-03-30T14:25:10.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 60167 14.87 MB/sec warmup 115 sec latency 67.311 ms 2024-03-30T14:25:10.472 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:10 smithi031 ceph-mon[24911]: pgmap v296: 97 pgs: 97 active+clean; 314 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-03-30T14:25:10.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:10 smithi193 ceph-mon[26933]: pgmap v296: 97 pgs: 97 active+clean; 314 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-03-30T14:25:11.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 60707 14.89 MB/sec warmup 116 sec latency 41.158 ms 2024-03-30T14:25:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 61099 14.82 MB/sec warmup 117 sec latency 28.585 ms 2024-03-30T14:25:12.674 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:12 smithi031 ceph-mon[24911]: pgmap v297: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.1 MiB/s wr, 763 op/s 2024-03-30T14:25:12.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:12 smithi193 ceph-mon[26933]: pgmap v297: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.1 MiB/s wr, 763 op/s 2024-03-30T14:25:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: 5 61582 14.78 MB/sec warmup 118 sec latency 57.567 ms 2024-03-30T14:25:13.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:25:13.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:13 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:25:13.679 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:25:13.680 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:13 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:25:14.104 INFO:teuthology.orchestra.run.smithi031.stdout: 5 62074 14.73 MB/sec warmup 119 sec latency 75.796 ms 2024-03-30T14:25:14.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:14 smithi031 ceph-mon[24911]: pgmap v298: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 656 op/s 2024-03-30T14:25:14.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:14.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:14.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:14.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:25:14.685 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:14 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr fail", "who": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:25:14.685 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:14 smithi031 ceph-mon[24911]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T14:25:14.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:14 smithi193 ceph-mon[26933]: pgmap v298: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 656 op/s 2024-03-30T14:25:14.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:14.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:14.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' 2024-03-30T14:25:14.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:25:14.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:14 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr fail", "who": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:25:14.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:14 smithi193 ceph-mon[26933]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T14:25:15.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: Upgrade: Need to upgrade myself (mgr.smithi031.lkrchi) 2024-03-30T14:25:15.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: Failing over to other MGR 2024-03-30T14:25:15.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: Standby manager daemon smithi031.lkrchi started 2024-03-30T14:25:15.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "mgr fail", "who": "smithi031.lkrchi"}]': finished 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: mgrmap e23: smithi193.ikzlnb(active, starting, since 1.00292s), standbys: smithi031.lkrchi 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr metadata", "who": "smithi193.ikzlnb", "id": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:25:15.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: Manager daemon smithi193.ikzlnb is now available 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/trash_purge_schedule"}]: dispatch 2024-03-30T14:25:15.501 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:15 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/trash_purge_schedule"}]: dispatch 2024-03-30T14:25:15.692 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: Upgrade: Need to upgrade myself (mgr.smithi031.lkrchi) 2024-03-30T14:25:15.692 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: Failing over to other MGR 2024-03-30T14:25:15.692 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: Standby manager daemon smithi031.lkrchi started 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14202 172.21.15.31:0/1093839133' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "mgr fail", "who": "smithi031.lkrchi"}]': finished 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: mgrmap e23: smithi193.ikzlnb(active, starting, since 1.00292s), standbys: smithi031.lkrchi 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr metadata", "who": "smithi193.ikzlnb", "id": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:25:15.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: Manager daemon smithi193.ikzlnb is now available 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/trash_purge_schedule"}]: dispatch 2024-03-30T14:25:15.694 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:15 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi193.ikzlnb/trash_purge_schedule"}]: dispatch 2024-03-30T14:25:16.104 INFO:teuthology.orchestra.run.smithi031.stdout: 5 63149 9.46 MB/sec execute 1 sec latency 66.197 ms 2024-03-30T14:25:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: 5 63666 16.86 MB/sec execute 2 sec latency 36.628 ms 2024-03-30T14:25:17.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:17 smithi031 ceph-mon[24911]: Deploying cephadm binary to smithi193 2024-03-30T14:25:17.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:17 smithi031 ceph-mon[24911]: Deploying cephadm binary to smithi031 2024-03-30T14:25:17.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:17 smithi031 ceph-mon[24911]: mgrmap e24: smithi193.ikzlnb(active, since 2s), standbys: smithi031.lkrchi 2024-03-30T14:25:17.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:17 smithi031 ceph-mon[24911]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:17.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:17 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:17.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:17 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:17.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:17 smithi193 ceph-mon[26933]: Deploying cephadm binary to smithi193 2024-03-30T14:25:17.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:17 smithi193 ceph-mon[26933]: Deploying cephadm binary to smithi031 2024-03-30T14:25:17.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:17 smithi193 ceph-mon[26933]: mgrmap e24: smithi193.ikzlnb(active, since 2s), standbys: smithi031.lkrchi 2024-03-30T14:25:17.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:17 smithi193 ceph-mon[26933]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:17.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:17 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:17.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:17 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:18.104 INFO:teuthology.orchestra.run.smithi031.stdout: 5 64094 15.68 MB/sec execute 3 sec latency 43.645 ms 2024-03-30T14:25:18.454 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:25:18.482 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:18 smithi031 ceph-mon[24911]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:18.483 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:18 smithi031 ceph-mon[24911]: mgrmap e25: smithi193.ikzlnb(active, since 3s), standbys: smithi031.lkrchi 2024-03-30T14:25:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:18 smithi193 ceph-mon[26933]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:18.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:18 smithi193 ceph-mon[26933]: mgrmap e25: smithi193.ikzlnb(active, since 3s), standbys: smithi031.lkrchi 2024-03-30T14:25:19.104 INFO:teuthology.orchestra.run.smithi031.stdout: 5 64416 13.74 MB/sec execute 4 sec latency 41.036 ms 2024-03-30T14:25:19.492 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:19 smithi031 ceph-mon[24911]: [30/Mar/2024:14:25:17] ENGINE Bus STARTING 2024-03-30T14:25:19.493 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:19 smithi031 ceph-mon[24911]: [30/Mar/2024:14:25:17] ENGINE Serving on http://172.21.15.193:8765 2024-03-30T14:25:19.493 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:19 smithi031 ceph-mon[24911]: [30/Mar/2024:14:25:17] ENGINE Serving on https://172.21.15.193:7150 2024-03-30T14:25:19.493 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:19 smithi031 ceph-mon[24911]: [30/Mar/2024:14:25:17] ENGINE Bus STARTED 2024-03-30T14:25:19.493 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:19 smithi031 ceph-mon[24911]: mgrmap e26: smithi193.ikzlnb(active, since 4s), standbys: smithi031.lkrchi 2024-03-30T14:25:19.493 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:19 smithi031 ceph-mon[24911]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:19.567 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:19 smithi193 ceph-mon[26933]: [30/Mar/2024:14:25:17] ENGINE Bus STARTING 2024-03-30T14:25:19.567 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:19 smithi193 ceph-mon[26933]: [30/Mar/2024:14:25:17] ENGINE Serving on http://172.21.15.193:8765 2024-03-30T14:25:19.568 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:19 smithi193 ceph-mon[26933]: [30/Mar/2024:14:25:17] ENGINE Serving on https://172.21.15.193:7150 2024-03-30T14:25:19.568 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:19 smithi193 ceph-mon[26933]: [30/Mar/2024:14:25:17] ENGINE Bus STARTED 2024-03-30T14:25:19.568 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:19 smithi193 ceph-mon[26933]: mgrmap e26: smithi193.ikzlnb(active, since 4s), standbys: smithi031.lkrchi 2024-03-30T14:25:19.568 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:19 smithi193 ceph-mon[26933]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (7m) 2m ago 8m 24.8M - ba2b418f427c 9fc4502dff1a 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (8m) 2m ago 8m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (7m) 18s ago 7m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (6m) 2m ago 8m 53.9M - 8.3.5 dad864ee21e9 c132179ebde6 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (2m) 2m ago 2m 15.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (2m) 18s ago 2m 12.6M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:9283 running (9m) 2m ago 9m 460M - 17.2.0 e1d6a67b021e 330ec216c990 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (22s) 18s ago 7m 49.6M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 038883e261d6 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (9m) 2m ago 9m 44.1M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (7m) 18s ago 7m 41.6M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (2m) 2m ago 2m 50.9M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (2m) 18s ago 2m 71.3M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:25:19.595 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (8m) 2m ago 8m 17.7M - 1dbe0e931976 a5b6d8e0efa4 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (7m) 18s ago 7m 17.2M - 1dbe0e931976 3f7d1e5ad9be 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (7m) 2m ago 7m 42.1M 3217M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (6m) 2m ago 6m 43.1M 3217M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (6m) 2m ago 6m 42.2M 3217M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (5m) 2m ago 5m 40.7M 3217M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (5m) 18s ago 5m 229M 3984M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (4m) 18s ago 4m 225M 3984M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (4m) 18s ago 4m 212M 3984M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (4m) 18s ago 4m 306M 3984M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:25:19.596 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (7m) 2m ago 7m 62.1M - 514e6a882f6e 15161d5c677a 2024-03-30T14:25:20.104 INFO:teuthology.orchestra.run.smithi031.stdout: 5 64744 11.76 MB/sec execute 5 sec latency 42.689 ms 2024-03-30T14:25:20.183 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:25:20.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:25:20.183 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-30T14:25:20.184 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:25:20.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:25:20.185 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:25:20.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:20 smithi031 ceph-mon[24911]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:20.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:20 smithi031 ceph-mon[24911]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:20.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:20 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/1442921395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:25:20.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:20 smithi193 ceph-mon[26933]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:20.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:20 smithi193 ceph-mon[26933]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:20.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:20 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/1442921395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "1/23 daemons upgraded", 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "", 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:25:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:25:21.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 65145 11.32 MB/sec execute 6 sec latency 107.750 ms 2024-03-30T14:25:21.263 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:25:21.281 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:21 smithi031 ceph-mon[24911]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:21.281 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:21 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2370362120' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:25:21.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:21 smithi193 ceph-mon[26933]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:21.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:21 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2370362120' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:25:22.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 65580 11.01 MB/sec execute 7 sec latency 46.104 ms 2024-03-30T14:25:22.296 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:22 smithi031 ceph-mon[24911]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:22.296 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:22 smithi031 ceph-mon[24911]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:22 smithi193 ceph-mon[26933]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:22.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:22 smithi193 ceph-mon[26933]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:25:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 65972 10.78 MB/sec execute 8 sec latency 124.972 ms 2024-03-30T14:25:23.899 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:23 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:23.899 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:23 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:23.899 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:23 smithi031 ceph-mon[24911]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 509 op/s 2024-03-30T14:25:24.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:23 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:24.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:23 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:24.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:23 smithi193 ceph-mon[26933]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 509 op/s 2024-03-30T14:25:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 66542 11.18 MB/sec execute 9 sec latency 39.785 ms 2024-03-30T14:25:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 66947 11.81 MB/sec execute 10 sec latency 41.822 ms 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: Standby manager daemon smithi031.lkrchi restarted 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: Standby manager daemon smithi031.lkrchi started 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/crt"}]: dispatch 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/key"}]: dispatch 2024-03-30T14:25:25.155 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:24 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:25:25.328 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:25.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:25.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: Standby manager daemon smithi031.lkrchi restarted 2024-03-30T14:25:25.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: Standby manager daemon smithi031.lkrchi started 2024-03-30T14:25:25.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/crt"}]: dispatch 2024-03-30T14:25:25.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:25:25.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/key"}]: dispatch 2024-03-30T14:25:25.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:24 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/1486231883' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:25:26.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 67543 12.64 MB/sec execute 11 sec latency 22.469 ms 2024-03-30T14:25:26.162 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:26 smithi193 ceph-mon[26933]: mgrmap e27: smithi193.ikzlnb(active, since 10s), standbys: smithi031.lkrchi 2024-03-30T14:25:26.162 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:26 smithi193 ceph-mon[26933]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 3.4 MiB/s wr, 395 op/s 2024-03-30T14:25:26.166 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:26 smithi031 ceph-mon[24911]: mgrmap e27: smithi193.ikzlnb(active, since 10s), standbys: smithi031.lkrchi 2024-03-30T14:25:26.166 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:26 smithi031 ceph-mon[24911]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 3.4 MiB/s wr, 395 op/s 2024-03-30T14:25:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 67995 12.56 MB/sec execute 12 sec latency 24.992 ms 2024-03-30T14:25:28.105 INFO:teuthology.orchestra.run.smithi031.stdout: 5 68518 12.42 MB/sec execute 13 sec latency 39.934 ms 2024-03-30T14:25:28.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 633 op/s 2024-03-30T14:25:28.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:28.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:28.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.441 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:28 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 633 op/s 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:28.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:28 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:29.111 INFO:teuthology.orchestra.run.smithi031.stdout: 5 69053 12.22 MB/sec execute 14 sec latency 133.531 ms 2024-03-30T14:25:29.454 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:29 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi193 to 2704M 2024-03-30T14:25:29.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:29 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi193 to 2704M 2024-03-30T14:25:30.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 69730 12.57 MB/sec execute 15 sec latency 32.680 ms 2024-03-30T14:25:30.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:30 smithi031 ceph-mon[24911]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 577 op/s 2024-03-30T14:25:30.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:30 smithi193 ceph-mon[26933]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 577 op/s 2024-03-30T14:25:31.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 70293 12.74 MB/sec execute 16 sec latency 32.816 ms 2024-03-30T14:25:31.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:31 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:31.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:31 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:31.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:31 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:31.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:31 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:31.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:31 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:31.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:31 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:32.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 70794 13.31 MB/sec execute 17 sec latency 32.913 ms 2024-03-30T14:25:32.487 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:32 smithi031 ceph-mon[24911]: pgmap v11: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 577 op/s 2024-03-30T14:25:32.578 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:32 smithi193 ceph-mon[26933]: pgmap v11: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 577 op/s 2024-03-30T14:25:33.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 71220 13.37 MB/sec execute 18 sec latency 37.546 ms 2024-03-30T14:25:33.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:33.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:33.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: Adjusting osd_memory_target on smithi031 to 1937M 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:25:33.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:33 smithi031 ceph-mon[24911]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:25:33.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: Adjusting osd_memory_target on smithi031 to 1937M 2024-03-30T14:25:33.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:33.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:25:33.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:25:33.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:25:33.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:33 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:25:34.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 71573 12.99 MB/sec execute 19 sec latency 36.265 ms 2024-03-30T14:25:34.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:25:34.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: pgmap v12: 97 pgs: 97 active+clean; 326 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 908 op/s 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:34 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:25:34.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:25:34.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: pgmap v12: 97 pgs: 97 active+clean; 326 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 908 op/s 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:34.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:34 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:35.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 72117 12.87 MB/sec execute 20 sec latency 38.775 ms 2024-03-30T14:25:35.521 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:35 smithi031 ceph-mon[24911]: Reconfiguring alertmanager.smithi031 (dependencies changed)... 2024-03-30T14:25:35.521 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:35 smithi031 ceph-mon[24911]: Reconfiguring daemon alertmanager.smithi031 on smithi031 2024-03-30T14:25:35.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:35 smithi193 ceph-mon[26933]: Reconfiguring alertmanager.smithi031 (dependencies changed)... 2024-03-30T14:25:35.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:35 smithi193 ceph-mon[26933]: Reconfiguring daemon alertmanager.smithi031 on smithi031 2024-03-30T14:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 72706 12.78 MB/sec execute 21 sec latency 44.493 ms 2024-03-30T14:25:36.529 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:36 smithi031 ceph-mon[24911]: pgmap v13: 97 pgs: 97 active+clean; 326 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 613 op/s 2024-03-30T14:25:36.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:36 smithi193 ceph-mon[26933]: pgmap v13: 97 pgs: 97 active+clean; 326 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 613 op/s 2024-03-30T14:25:37.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 73178 12.76 MB/sec execute 22 sec latency 31.254 ms 2024-03-30T14:25:38.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 73755 12.83 MB/sec execute 23 sec latency 29.467 ms 2024-03-30T14:25:38.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:38 smithi031 ceph-mon[24911]: pgmap v14: 97 pgs: 97 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.2 MiB/s wr, 904 op/s 2024-03-30T14:25:38.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:38 smithi193 ceph-mon[26933]: pgmap v14: 97 pgs: 97 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.2 MiB/s wr, 904 op/s 2024-03-30T14:25:39.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 74276 13.34 MB/sec execute 24 sec latency 31.760 ms 2024-03-30T14:25:40.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 74875 13.53 MB/sec execute 25 sec latency 28.709 ms 2024-03-30T14:25:40.582 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:40 smithi031 ceph-mon[24911]: pgmap v15: 97 pgs: 97 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 621 op/s 2024-03-30T14:25:40.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:40 smithi193 ceph-mon[26933]: pgmap v15: 97 pgs: 97 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 621 op/s 2024-03-30T14:25:41.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 75252 13.27 MB/sec execute 26 sec latency 34.214 ms 2024-03-30T14:25:42.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 75807 13.13 MB/sec execute 27 sec latency 37.067 ms 2024-03-30T14:25:42.615 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:42 smithi031 ceph-mon[24911]: pgmap v16: 97 pgs: 97 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 621 op/s 2024-03-30T14:25:42.828 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:42 smithi193 ceph-mon[26933]: pgmap v16: 97 pgs: 97 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 621 op/s 2024-03-30T14:25:43.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 76355 13.13 MB/sec execute 28 sec latency 36.353 ms 2024-03-30T14:25:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 77015 13.23 MB/sec execute 29 sec latency 32.138 ms 2024-03-30T14:25:44.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:44 smithi193 ceph-mon[26933]: pgmap v17: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 922 op/s 2024-03-30T14:25:44.638 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:44 smithi031 ceph-mon[24911]: pgmap v17: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 922 op/s 2024-03-30T14:25:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 77533 13.43 MB/sec execute 30 sec latency 34.664 ms 2024-03-30T14:25:46.038 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:45 smithi031 ceph-mon[24911]: pgmap v18: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.4 MiB/s wr, 591 op/s 2024-03-30T14:25:46.038 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:45 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 78008 13.53 MB/sec execute 31 sec latency 41.144 ms 2024-03-30T14:25:46.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:45 smithi193 ceph-mon[26933]: pgmap v18: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.4 MiB/s wr, 591 op/s 2024-03-30T14:25:46.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:45 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:47.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 78492 13.58 MB/sec execute 32 sec latency 35.371 ms 2024-03-30T14:25:48.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 78845 13.36 MB/sec execute 33 sec latency 66.951 ms 2024-03-30T14:25:48.306 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:48 smithi031 ceph-mon[24911]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 911 op/s 2024-03-30T14:25:48.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:48 smithi193 ceph-mon[26933]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 911 op/s 2024-03-30T14:25:49.112 INFO:teuthology.orchestra.run.smithi031.stdout: 5 79349 13.24 MB/sec execute 34 sec latency 36.017 ms 2024-03-30T14:25:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 79840 13.15 MB/sec execute 35 sec latency 49.480 ms 2024-03-30T14:25:50.365 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:50 smithi031 ceph-mon[24911]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.7 MiB/s wr, 620 op/s 2024-03-30T14:25:50.365 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:50 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:50.365 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:50 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:50.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:50 smithi193 ceph-mon[26933]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.7 MiB/s wr, 620 op/s 2024-03-30T14:25:50.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:50 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:50.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:50 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:51.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 80314 13.13 MB/sec execute 36 sec latency 40.792 ms 2024-03-30T14:25:51.364 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:51 smithi031 ceph-mon[24911]: Reconfiguring grafana.smithi031 (dependencies changed)... 2024-03-30T14:25:51.364 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:51 smithi031 ceph-mon[24911]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T14:25:51.364 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:51 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:51.364 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:51 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:51.364 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:51 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:25:51.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:51 smithi193 ceph-mon[26933]: Reconfiguring grafana.smithi031 (dependencies changed)... 2024-03-30T14:25:51.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:51 smithi193 ceph-mon[26933]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T14:25:51.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:51 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:51.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:51 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:25:51.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:51 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:25:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:25:52.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 80920 13.25 MB/sec execute 37 sec latency 32.984 ms 2024-03-30T14:25:52.372 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:52 smithi031 ceph-mon[24911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:25:52.373 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:52 smithi031 ceph-mon[24911]: Reconfiguring daemon grafana.smithi031 on smithi031 2024-03-30T14:25:52.373 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:52 smithi031 ceph-mon[24911]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.7 MiB/s wr, 620 op/s 2024-03-30T14:25:52.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:52 smithi193 ceph-mon[26933]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:25:52.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:52 smithi193 ceph-mon[26933]: Reconfiguring daemon grafana.smithi031 on smithi031 2024-03-30T14:25:52.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:52 smithi193 ceph-mon[26933]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.7 MiB/s wr, 620 op/s 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 starting - - - - 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (9m) 22s ago 9m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (8m) 30s ago 8m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (7m) 22s ago 8m 56.0M - 8.3.5 dad864ee21e9 c132179ebde6 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (3m) 22s ago 3m 42.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (3m) 30s ago 3m 12.9M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:9283 running (10m) 22s ago 10m 337M - 17.2.0 e1d6a67b021e 330ec216c990 2024-03-30T14:25:52.981 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (55s) 30s ago 8m 492M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 038883e261d6 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (10m) 22s ago 10m 51.0M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (8m) 30s ago 8m 42.2M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (2m) 22s ago 2m 236M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (2m) 30s ago 2m 76.2M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (8m) 22s ago 8m 16.0M - 1.3.1 1dbe0e931976 a5b6d8e0efa4 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (8m) 30s ago 8m 17.4M - 1.3.1 1dbe0e931976 3f7d1e5ad9be 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (7m) 22s ago 7m 292M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (7m) 22s ago 7m 224M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (6m) 22s ago 6m 203M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (6m) 22s ago 6m 234M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (5m) 30s ago 5m 246M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (5m) 30s ago 5m 283M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (5m) 30s ago 5m 249M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:25:52.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (4m) 30s ago 4m 330M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:25:52.983 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (7m) 22s ago 7m 62.2M - 2.33.4 514e6a882f6e 15161d5c677a 2024-03-30T14:25:53.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 81451 13.50 MB/sec execute 38 sec latency 31.247 ms 2024-03-30T14:25:53.387 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:53 smithi031 ceph-mon[24911]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:53.387 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:53 smithi031 ceph-mon[24911]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:53.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:53 smithi193 ceph-mon[26933]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:53.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:53 smithi193 ceph-mon[26933]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:25:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:25:53.617 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:25:54.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 81901 13.49 MB/sec execute 39 sec latency 33.228 ms 2024-03-30T14:25:54.233 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:25:54.233 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:25:54.234 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:25:54.234 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:25:54.234 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:25:54.234 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "1/23 daemons upgraded", 2024-03-30T14:25:54.234 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "", 2024-03-30T14:25:54.234 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:25:54.234 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:25:54.393 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:54 smithi031 ceph-mon[24911]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:54.393 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:54 smithi031 ceph-mon[24911]: pgmap v22: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 906 op/s 2024-03-30T14:25:54.393 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:54 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3133960417' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:25:54.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:54 smithi193 ceph-mon[26933]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:54.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:54 smithi193 ceph-mon[26933]: pgmap v22: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 906 op/s 2024-03-30T14:25:54.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:54 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3133960417' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:25:54.970 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:25:55.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 82204 13.33 MB/sec execute 40 sec latency 44.454 ms 2024-03-30T14:25:55.410 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:55 smithi031 ceph-mon[24911]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:55.413 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:55 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/349503730' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:25:55.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:55 smithi193 ceph-mon[26933]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:25:55.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:55 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/349503730' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:25:56.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 82623 13.25 MB/sec execute 41 sec latency 86.555 ms 2024-03-30T14:25:56.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:56 smithi031 ceph-mon[24911]: pgmap v23: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 5.7 MiB/s wr, 606 op/s 2024-03-30T14:25:56.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:56 smithi193 ceph-mon[26933]: pgmap v23: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 5.7 MiB/s wr, 606 op/s 2024-03-30T14:25:57.113 INFO:teuthology.orchestra.run.smithi031.stdout: 5 83326 13.20 MB/sec execute 42 sec latency 24.822 ms 2024-03-30T14:25:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: 5 84074 13.36 MB/sec execute 43 sec latency 37.910 ms 2024-03-30T14:25:58.431 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:25:58 smithi031 ceph-mon[24911]: pgmap v24: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 884 op/s 2024-03-30T14:25:58.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:25:58 smithi193 ceph-mon[26933]: pgmap v24: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 884 op/s 2024-03-30T14:25:59.114 INFO:teuthology.orchestra.run.smithi031.stdout: 5 84760 13.62 MB/sec execute 44 sec latency 32.944 ms 2024-03-30T14:26:00.114 INFO:teuthology.orchestra.run.smithi031.stdout: 5 85264 13.69 MB/sec execute 45 sec latency 43.428 ms 2024-03-30T14:26:00.440 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:00 smithi031 ceph-mon[24911]: pgmap v25: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 564 op/s 2024-03-30T14:26:00.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:00 smithi193 ceph-mon[26933]: pgmap v25: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 564 op/s 2024-03-30T14:26:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 85624 13.63 MB/sec execute 46 sec latency 43.058 ms 2024-03-30T14:26:01.446 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:01 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:01.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:01 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:02.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 85975 13.45 MB/sec execute 47 sec latency 48.230 ms 2024-03-30T14:26:02.452 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:02 smithi031 ceph-mon[24911]: pgmap v26: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 564 op/s 2024-03-30T14:26:02.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:02 smithi193 ceph-mon[26933]: pgmap v26: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 564 op/s 2024-03-30T14:26:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 86444 13.37 MB/sec execute 48 sec latency 33.500 ms 2024-03-30T14:26:04.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 86916 13.29 MB/sec execute 49 sec latency 35.211 ms 2024-03-30T14:26:04.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:04 smithi031 ceph-mon[24911]: pgmap v27: 97 pgs: 97 active+clean; 335 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 906 op/s 2024-03-30T14:26:04.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:04 smithi193 ceph-mon[26933]: pgmap v27: 97 pgs: 97 active+clean; 335 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 906 op/s 2024-03-30T14:26:05.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 87530 13.37 MB/sec execute 50 sec latency 49.126 ms 2024-03-30T14:26:05.992 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:05 smithi031 ceph-mon[24911]: pgmap v28: 97 pgs: 97 active+clean; 335 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.9 MiB/s wr, 620 op/s 2024-03-30T14:26:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 88168 13.51 MB/sec execute 51 sec latency 25.082 ms 2024-03-30T14:26:06.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:05 smithi193 ceph-mon[26933]: pgmap v28: 97 pgs: 97 active+clean; 335 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.9 MiB/s wr, 620 op/s 2024-03-30T14:26:07.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 88670 13.64 MB/sec execute 52 sec latency 30.055 ms 2024-03-30T14:26:08.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 89174 13.62 MB/sec execute 53 sec latency 30.639 ms 2024-03-30T14:26:08.406 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:08 smithi031 ceph-mon[24911]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 929 op/s 2024-03-30T14:26:08.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:08 smithi193 ceph-mon[26933]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 929 op/s 2024-03-30T14:26:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 89583 13.49 MB/sec execute 54 sec latency 39.981 ms 2024-03-30T14:26:10.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 90186 13.43 MB/sec execute 55 sec latency 25.578 ms 2024-03-30T14:26:10.169 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:10 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:10.169 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:10 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:10.169 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:10 smithi031 ceph-mon[24911]: Reconfiguring node-exporter.smithi031 (dependencies changed)... 2024-03-30T14:26:10.170 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:10 smithi031 ceph-mon[24911]: Reconfiguring daemon node-exporter.smithi031 on smithi031 2024-03-30T14:26:10.170 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:10 smithi031 ceph-mon[24911]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 651 op/s 2024-03-30T14:26:10.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:10 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:10.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:10 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:10.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:10 smithi193 ceph-mon[26933]: Reconfiguring node-exporter.smithi031 (dependencies changed)... 2024-03-30T14:26:10.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:10 smithi193 ceph-mon[26933]: Reconfiguring daemon node-exporter.smithi031 on smithi031 2024-03-30T14:26:10.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:10 smithi193 ceph-mon[26933]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 651 op/s 2024-03-30T14:26:11.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 90793 13.49 MB/sec execute 56 sec latency 27.476 ms 2024-03-30T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: 5 91365 13.48 MB/sec execute 57 sec latency 31.093 ms 2024-03-30T14:26:12.455 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:12 smithi031 ceph-mon[24911]: pgmap v31: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 651 op/s 2024-03-30T14:26:12.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:12 smithi193 ceph-mon[26933]: pgmap v31: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 651 op/s 2024-03-30T14:26:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 91800 13.57 MB/sec execute 58 sec latency 43.948 ms 2024-03-30T14:26:14.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 92314 13.67 MB/sec execute 59 sec latency 40.918 ms 2024-03-30T14:26:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:14 smithi031 ceph-mon[24911]: pgmap v32: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.7 MiB/s wr, 971 op/s 2024-03-30T14:26:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:14 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:14 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:14 smithi031 ceph-mon[24911]: Reconfiguring prometheus.smithi031 (dependencies changed)... 2024-03-30T14:26:14.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:14 smithi193 ceph-mon[26933]: pgmap v32: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.7 MiB/s wr, 971 op/s 2024-03-30T14:26:14.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:14 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:14.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:14 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:14.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:14 smithi193 ceph-mon[26933]: Reconfiguring prometheus.smithi031 (dependencies changed)... 2024-03-30T14:26:15.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 92759 13.66 MB/sec execute 60 sec latency 29.109 ms 2024-03-30T14:26:15.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:15 smithi031 ceph-mon[24911]: Reconfiguring daemon prometheus.smithi031 on smithi031 2024-03-30T14:26:15.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:15 smithi193 ceph-mon[26933]: Reconfiguring daemon prometheus.smithi031 on smithi031 2024-03-30T14:26:16.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 93220 13.56 MB/sec execute 61 sec latency 53.862 ms 2024-03-30T14:26:16.522 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:16 smithi031 ceph-mon[24911]: pgmap v33: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.1 MiB/s wr, 629 op/s 2024-03-30T14:26:16.522 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:16 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:16.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:16 smithi193 ceph-mon[26933]: pgmap v33: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.1 MiB/s wr, 629 op/s 2024-03-30T14:26:16.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:16 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 93832 13.53 MB/sec execute 62 sec latency 176.205 ms 2024-03-30T14:26:18.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 94357 13.51 MB/sec execute 63 sec latency 196.964 ms 2024-03-30T14:26:18.397 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:18 smithi031 ceph-mon[24911]: pgmap v34: 97 pgs: 97 active+clean; 336 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 920 op/s 2024-03-30T14:26:18.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:18 smithi193 ceph-mon[26933]: pgmap v34: 97 pgs: 97 active+clean; 336 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 920 op/s 2024-03-30T14:26:19.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 94930 13.52 MB/sec execute 64 sec latency 68.495 ms 2024-03-30T14:26:20.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 95415 13.61 MB/sec execute 65 sec latency 32.422 ms 2024-03-30T14:26:20.422 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:20 smithi031 ceph-mon[24911]: pgmap v35: 97 pgs: 97 active+clean; 336 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 612 op/s 2024-03-30T14:26:20.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:20 smithi193 ceph-mon[26933]: pgmap v35: 97 pgs: 97 active+clean; 336 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 612 op/s 2024-03-30T14:26:21.118 INFO:teuthology.orchestra.run.smithi031.stdout: 5 95735 13.63 MB/sec execute 66 sec latency 40.521 ms 2024-03-30T14:26:22.117 INFO:teuthology.orchestra.run.smithi031.stdout: 5 96110 13.60 MB/sec execute 67 sec latency 34.624 ms 2024-03-30T14:26:22.440 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:22 smithi031 ceph-mon[24911]: pgmap v36: 97 pgs: 97 active+clean; 336 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 612 op/s 2024-03-30T14:26:22.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:22 smithi193 ceph-mon[26933]: pgmap v36: 97 pgs: 97 active+clean; 336 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 612 op/s 2024-03-30T14:26:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: 5 96422 13.48 MB/sec execute 68 sec latency 54.057 ms 2024-03-30T14:26:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: 5 96945 13.44 MB/sec execute 69 sec latency 54.234 ms 2024-03-30T14:26:24.459 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:24 smithi031 ceph-mon[24911]: pgmap v37: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 906 op/s 2024-03-30T14:26:24.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:24 smithi193 ceph-mon[26933]: pgmap v37: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 906 op/s 2024-03-30T14:26:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: 5 97389 13.34 MB/sec execute 70 sec latency 71.749 ms 2024-03-30T14:26:25.547 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:26:25.979 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:25 smithi031 ceph-mon[24911]: pgmap v38: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.4 MiB/s wr, 585 op/s 2024-03-30T14:26:26.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 97637 13.22 MB/sec execute 71 sec latency 42.893 ms 2024-03-30T14:26:26.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:25 smithi193 ceph-mon[26933]: pgmap v38: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.4 MiB/s wr, 585 op/s 2024-03-30T14:26:26.619 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:26:26.620 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 starting - - - - 2024-03-30T14:26:26.620 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (9m) 55s ago 9m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:26:26.620 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (9m) 63s ago 9m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:26:26.620 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 starting - - - - 2024-03-30T14:26:26.620 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (3m) 55s ago 3m 42.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:26:26.620 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (3m) 63s ago 3m 12.9M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:26:26.620 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:9283 running (10m) 55s ago 10m 337M - 17.2.0 e1d6a67b021e 330ec216c990 2024-03-30T14:26:26.621 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (89s) 63s ago 8m 492M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 038883e261d6 2024-03-30T14:26:26.621 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (11m) 55s ago 11m 51.0M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:26:26.621 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (8m) 63s ago 8m 42.2M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:26:26.621 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (3m) 55s ago 3m 236M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:26:26.621 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (3m) 63s ago 3m 76.2M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:26:26.621 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 starting - - - - 2024-03-30T14:26:26.621 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (8m) 63s ago 8m 17.4M - 1.3.1 1dbe0e931976 3f7d1e5ad9be 2024-03-30T14:26:26.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (8m) 55s ago 8m 292M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:26:26.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (7m) 55s ago 7m 224M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:26:26.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (7m) 55s ago 7m 203M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:26:26.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (6m) 55s ago 6m 234M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:26:26.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (6m) 63s ago 6m 246M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:26:26.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (6m) 63s ago 6m 283M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:26:26.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (5m) 63s ago 5m 249M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:26:26.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (5m) 63s ago 5m 330M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:26:26.623 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (8m) 55s ago 8m 62.2M - 2.33.4 514e6a882f6e 15161d5c677a 2024-03-30T14:26:26.990 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:26 smithi031 ceph-mon[24911]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:26.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:26 smithi031 ceph-mon[24911]: from='client.24513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 97961 13.17 MB/sec execute 72 sec latency 54.002 ms 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:26:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:26:27.255 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:26:27.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:26 smithi193 ceph-mon[26933]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:27.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:26 smithi193 ceph-mon[26933]: from='client.24513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:27.755 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:26:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:26:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:26:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:26:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:26:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "1/23 daemons upgraded", 2024-03-30T14:26:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "", 2024-03-30T14:26:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:26:27.756 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:26:28.014 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:27 smithi031 ceph-mon[24911]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:28.014 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:27 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/4166024823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:26:28.014 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:27 smithi031 ceph-mon[24911]: pgmap v39: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 832 op/s 2024-03-30T14:26:28.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 98329 13.13 MB/sec execute 73 sec latency 81.974 ms 2024-03-30T14:26:28.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:27 smithi193 ceph-mon[26933]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:28.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:27 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/4166024823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:26:28.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:27 smithi193 ceph-mon[26933]: pgmap v39: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 832 op/s 2024-03-30T14:26:28.422 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:26:29.030 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:28 smithi031 ceph-mon[24911]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:29.030 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:28 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2053326063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:26:29.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 98754 13.15 MB/sec execute 74 sec latency 31.450 ms 2024-03-30T14:26:29.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:28 smithi193 ceph-mon[26933]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:26:29.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:28 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2053326063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:26:29.944 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:29 smithi031 ceph-mon[24911]: pgmap v40: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.2 MiB/s wr, 540 op/s 2024-03-30T14:26:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 99172 13.21 MB/sec execute 75 sec latency 60.895 ms 2024-03-30T14:26:30.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:29 smithi193 ceph-mon[26933]: pgmap v40: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.2 MiB/s wr, 540 op/s 2024-03-30T14:26:31.071 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:30 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:31.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 99527 13.17 MB/sec execute 76 sec latency 50.214 ms 2024-03-30T14:26:31.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:30 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:32.086 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:31 smithi031 ceph-mon[24911]: pgmap v41: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.2 MiB/s wr, 540 op/s 2024-03-30T14:26:32.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 99903 13.13 MB/sec execute 77 sec latency 38.662 ms 2024-03-30T14:26:32.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:31 smithi193 ceph-mon[26933]: pgmap v41: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.2 MiB/s wr, 540 op/s 2024-03-30T14:26:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 100261 13.06 MB/sec execute 78 sec latency 28.466 ms 2024-03-30T14:26:34.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 100768 12.98 MB/sec execute 79 sec latency 62.076 ms 2024-03-30T14:26:34.331 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:34 smithi031 ceph-mon[24911]: pgmap v42: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 755 op/s 2024-03-30T14:26:34.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:34 smithi193 ceph-mon[26933]: pgmap v42: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 755 op/s 2024-03-30T14:26:35.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 101219 12.94 MB/sec execute 80 sec latency 40.320 ms 2024-03-30T14:26:36.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:35 smithi031 ceph-mon[24911]: pgmap v43: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 460 op/s 2024-03-30T14:26:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 101765 13.00 MB/sec execute 81 sec latency 43.645 ms 2024-03-30T14:26:36.329 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:35 smithi193 ceph-mon[26933]: pgmap v43: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 460 op/s 2024-03-30T14:26:37.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 102233 13.02 MB/sec execute 82 sec latency 40.545 ms 2024-03-30T14:26:38.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 102566 12.99 MB/sec execute 83 sec latency 28.424 ms 2024-03-30T14:26:38.402 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:38 smithi031 ceph-mon[24911]: pgmap v44: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 744 op/s 2024-03-30T14:26:38.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:38 smithi193 ceph-mon[26933]: pgmap v44: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 744 op/s 2024-03-30T14:26:39.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 102971 13.01 MB/sec execute 84 sec latency 40.762 ms 2024-03-30T14:26:39.680 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:39 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:39.680 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:39 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:39.680 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:39 smithi031 ceph-mon[24911]: Reconfiguring node-exporter.smithi193 (dependencies changed)... 2024-03-30T14:26:39.680 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:39 smithi031 ceph-mon[24911]: Reconfiguring daemon node-exporter.smithi193 on smithi193 2024-03-30T14:26:39.681 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:39 smithi031 ceph-mon[24911]: pgmap v45: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-03-30T14:26:39.829 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:39 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:39.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:39 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:39.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:39 smithi193 ceph-mon[26933]: Reconfiguring node-exporter.smithi193 (dependencies changed)... 2024-03-30T14:26:39.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:39 smithi193 ceph-mon[26933]: Reconfiguring daemon node-exporter.smithi193 on smithi193 2024-03-30T14:26:39.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:39 smithi193 ceph-mon[26933]: pgmap v45: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-03-30T14:26:40.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 103298 12.98 MB/sec execute 85 sec latency 58.765 ms 2024-03-30T14:26:41.119 INFO:teuthology.orchestra.run.smithi031.stdout: 5 103519 12.89 MB/sec execute 86 sec latency 37.337 ms 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi031.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi031.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:26:42.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi031.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi031.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi031.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi031.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:26:42.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:41 smithi031 ceph-mon[24911]: pgmap v46: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-03-30T14:26:42.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 103797 12.83 MB/sec execute 87 sec latency 54.458 ms 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi031.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi031.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:26:42.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi031.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi031.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi031.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi031.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:26:42.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:41 smithi193 ceph-mon[26933]: pgmap v46: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-03-30T14:26:43.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 104323 12.73 MB/sec execute 88 sec latency 45.347 ms 2024-03-30T14:26:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 104806 12.71 MB/sec execute 89 sec latency 36.498 ms 2024-03-30T14:26:44.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:44 smithi031 ceph-mon[24911]: pgmap v47: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.6 MiB/s wr, 707 op/s 2024-03-30T14:26:44.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:44 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:44.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:44 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:26:44.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:44 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:26:44.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:44 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:26:44.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:44 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:26:44.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:44 smithi193 ceph-mon[26933]: pgmap v47: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.6 MiB/s wr, 707 op/s 2024-03-30T14:26:44.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:44 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:44.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:44 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:26:44.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:44 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:26:44.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:44 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:26:44.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:44 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 105546 12.82 MB/sec execute 90 sec latency 61.729 ms 2024-03-30T14:26:45.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:45 smithi031 ceph-mon[24911]: Upgrade: Updating mgr.smithi031.lkrchi 2024-03-30T14:26:45.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:45 smithi031 ceph-mon[24911]: Deploying daemon mgr.smithi031.lkrchi on smithi031 2024-03-30T14:26:45.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:45 smithi193 ceph-mon[26933]: Upgrade: Updating mgr.smithi031.lkrchi 2024-03-30T14:26:45.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:45 smithi193 ceph-mon[26933]: Deploying daemon mgr.smithi031.lkrchi on smithi031 2024-03-30T14:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 106410 13.04 MB/sec execute 91 sec latency 19.857 ms 2024-03-30T14:26:46.540 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:46 smithi031 ceph-mon[24911]: pgmap v48: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 493 op/s 2024-03-30T14:26:46.540 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:46 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:46.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:46 smithi193 ceph-mon[26933]: pgmap v48: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 493 op/s 2024-03-30T14:26:46.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:46 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:47.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 106966 13.05 MB/sec execute 92 sec latency 60.335 ms 2024-03-30T14:26:48.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:47 smithi031 ceph-mon[24911]: mgrmap e28: smithi193.ikzlnb(active, since 92s), standbys: smithi031.lkrchi 2024-03-30T14:26:48.100 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:47 smithi031 ceph-mon[24911]: pgmap v49: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 836 op/s 2024-03-30T14:26:48.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 107391 13.04 MB/sec execute 93 sec latency 50.870 ms 2024-03-30T14:26:48.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:47 smithi193 ceph-mon[26933]: mgrmap e28: smithi193.ikzlnb(active, since 92s), standbys: smithi031.lkrchi 2024-03-30T14:26:48.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:47 smithi193 ceph-mon[26933]: pgmap v49: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 836 op/s 2024-03-30T14:26:49.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 107819 12.93 MB/sec execute 94 sec latency 49.016 ms 2024-03-30T14:26:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: 5 108188 12.89 MB/sec execute 95 sec latency 63.396 ms 2024-03-30T14:26:50.372 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:50 smithi031 ceph-mon[24911]: pgmap v50: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.1 MiB/s wr, 553 op/s 2024-03-30T14:26:50.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:50 smithi193 ceph-mon[26933]: pgmap v50: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.1 MiB/s wr, 553 op/s 2024-03-30T14:26:51.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 108620 12.93 MB/sec execute 96 sec latency 31.157 ms 2024-03-30T14:26:52.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 108994 12.86 MB/sec execute 97 sec latency 65.807 ms 2024-03-30T14:26:52.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:52 smithi031 ceph-mon[24911]: pgmap v51: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.1 MiB/s wr, 553 op/s 2024-03-30T14:26:52.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:52 smithi193 ceph-mon[26933]: pgmap v51: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.1 MiB/s wr, 553 op/s 2024-03-30T14:26:53.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 109474 12.89 MB/sec execute 98 sec latency 30.723 ms 2024-03-30T14:26:54.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 109755 12.86 MB/sec execute 99 sec latency 36.358 ms 2024-03-30T14:26:54.462 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:54 smithi031 ceph-mon[24911]: pgmap v52: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 817 op/s 2024-03-30T14:26:54.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:54 smithi193 ceph-mon[26933]: pgmap v52: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 817 op/s 2024-03-30T14:26:55.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 110145 12.85 MB/sec execute 100 sec latency 56.593 ms 2024-03-30T14:26:56.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 110413 12.82 MB/sec execute 101 sec latency 52.362 ms 2024-03-30T14:26:56.243 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:56 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:56.244 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:56 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:56.244 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:56 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:26:56.244 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:56 smithi031 ceph-mon[24911]: pgmap v53: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.8 MiB/s wr, 607 op/s 2024-03-30T14:26:56.244 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:56 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:56.244 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:56 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:56.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:56 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:56.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:56 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:56.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:56 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:26:56.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:56 smithi193 ceph-mon[26933]: pgmap v53: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.8 MiB/s wr, 607 op/s 2024-03-30T14:26:56.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:56 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:56.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:56 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:26:57.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 110637 12.75 MB/sec execute 102 sec latency 41.262 ms 2024-03-30T14:26:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 110922 12.70 MB/sec execute 103 sec latency 55.795 ms 2024-03-30T14:26:58.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:26:58 smithi031 ceph-mon[24911]: pgmap v54: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 815 op/s 2024-03-30T14:26:58.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:26:58 smithi193 ceph-mon[26933]: pgmap v54: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 815 op/s 2024-03-30T14:26:59.016 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:26:59.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 111223 12.60 MB/sec execute 104 sec latency 49.166 ms 2024-03-30T14:27:00.121 INFO:teuthology.orchestra.run.smithi031.stdout: 5 111530 12.53 MB/sec execute 105 sec latency 62.526 ms 2024-03-30T14:27:00.200 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 starting - - - - 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (10m) 89s ago 10m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (9m) 0s ago 9m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 starting - - - - 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (4m) 89s ago 4m 42.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (4m) 0s ago 4m 13.4M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 starting - - - - 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (2m) 0s ago 9m 509M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 038883e261d6 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (11m) 89s ago 11m 51.0M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (9m) 0s ago 9m 41.9M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (3m) 89s ago 3m 236M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (3m) 0s ago 3m 80.3M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:27:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 starting - - - - 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (20s) 0s ago 9m 10.2M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (8m) 89s ago 8m 292M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (8m) 89s ago 8m 224M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (7m) 89s ago 7m 203M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (7m) 89s ago 7m 234M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (7m) 0s ago 7m 384M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (6m) 0s ago 6m 422M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (6m) 0s ago 6m 367M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (5m) 0s ago 5m 488M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:27:00.202 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 starting - - - - 2024-03-30T14:27:00.458 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:00 smithi031 ceph-mon[24911]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:00.459 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:00 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:00.459 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:00 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:00.459 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:00 smithi031 ceph-mon[24911]: pgmap v55: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 472 op/s 2024-03-30T14:27:00.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:00 smithi193 ceph-mon[26933]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:00.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:00 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:00.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:00 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:00.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:00 smithi193 ceph-mon[26933]: pgmap v55: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 472 op/s 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:00.933 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:27:00.934 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:27:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 111745 12.44 MB/sec execute 106 sec latency 56.511 ms 2024-03-30T14:27:01.473 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:01 smithi031 ceph-mon[24911]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:01.473 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:01 smithi031 ceph-mon[24911]: from='client.24541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:01.474 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:01 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:01.474 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:01 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/906543036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "", 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading mgr daemons", 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:27:01.486 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:27:01.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:01 smithi193 ceph-mon[26933]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:01.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:01 smithi193 ceph-mon[26933]: from='client.24541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:01.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:01 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:01.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:01 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/906543036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:27:02.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 112160 12.47 MB/sec execute 107 sec latency 58.038 ms 2024-03-30T14:27:02.199 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:27:02.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:02 smithi031 ceph-mon[24911]: pgmap v56: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 472 op/s 2024-03-30T14:27:02.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:02 smithi031 ceph-mon[24911]: from='client.24547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:02.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:02 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/192773693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:27:02.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:02 smithi193 ceph-mon[26933]: pgmap v56: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 472 op/s 2024-03-30T14:27:02.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:02 smithi193 ceph-mon[26933]: from='client.24547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:02.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:02 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/192773693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:27:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 112702 12.48 MB/sec execute 108 sec latency 26.013 ms 2024-03-30T14:27:04.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 113192 12.52 MB/sec execute 109 sec latency 21.064 ms 2024-03-30T14:27:04.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:04 smithi031 ceph-mon[24911]: pgmap v57: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 654 op/s 2024-03-30T14:27:04.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:04 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:04.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:04 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:04.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:04 smithi193 ceph-mon[26933]: pgmap v57: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 654 op/s 2024-03-30T14:27:04.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:04 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:04.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:04 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:05.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 113738 12.56 MB/sec execute 110 sec latency 37.346 ms 2024-03-30T14:27:06.007 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:05 smithi031 ceph-mon[24911]: pgmap v58: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 391 op/s 2024-03-30T14:27:06.008 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:05 smithi031 ceph-mon[24911]: Standby manager daemon smithi031.lkrchi restarted 2024-03-30T14:27:06.008 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:05 smithi031 ceph-mon[24911]: Standby manager daemon smithi031.lkrchi started 2024-03-30T14:27:06.008 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:05 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/crt"}]: dispatch 2024-03-30T14:27:06.008 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:05 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:27:06.008 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:05 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/key"}]: dispatch 2024-03-30T14:27:06.008 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:05 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:27:06.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:05 smithi193 ceph-mon[26933]: pgmap v58: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 391 op/s 2024-03-30T14:27:06.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:05 smithi193 ceph-mon[26933]: Standby manager daemon smithi031.lkrchi restarted 2024-03-30T14:27:06.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:05 smithi193 ceph-mon[26933]: Standby manager daemon smithi031.lkrchi started 2024-03-30T14:27:06.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:05 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/crt"}]: dispatch 2024-03-30T14:27:06.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:05 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:27:06.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:05 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.lkrchi/key"}]: dispatch 2024-03-30T14:27:06.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:05 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:27:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 114225 12.58 MB/sec execute 111 sec latency 28.041 ms 2024-03-30T14:27:07.019 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:06 smithi031 ceph-mon[24911]: mgrmap e29: smithi193.ikzlnb(active, since 111s), standbys: smithi031.lkrchi 2024-03-30T14:27:07.019 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:06 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:07.019 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:06 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:07.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:06 smithi193 ceph-mon[26933]: mgrmap e29: smithi193.ikzlnb(active, since 111s), standbys: smithi031.lkrchi 2024-03-30T14:27:07.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:06 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:07.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:06 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:07.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 114689 12.55 MB/sec execute 112 sec latency 37.398 ms 2024-03-30T14:27:08.028 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:07 smithi031 ceph-mon[24911]: pgmap v59: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.3 MiB/s wr, 682 op/s 2024-03-30T14:27:08.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 115249 12.51 MB/sec execute 113 sec latency 26.348 ms 2024-03-30T14:27:08.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:07 smithi193 ceph-mon[26933]: pgmap v59: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.3 MiB/s wr, 682 op/s 2024-03-30T14:27:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: 5 115756 12.58 MB/sec execute 114 sec latency 37.047 ms 2024-03-30T14:27:10.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 116257 12.56 MB/sec execute 115 sec latency 40.465 ms 2024-03-30T14:27:10.561 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:10 smithi031 ceph-mon[24911]: pgmap v60: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 474 op/s 2024-03-30T14:27:10.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:10 smithi193 ceph-mon[26933]: pgmap v60: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 474 op/s 2024-03-30T14:27:11.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 116707 12.58 MB/sec execute 116 sec latency 27.861 ms 2024-03-30T14:27:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 117383 12.65 MB/sec execute 117 sec latency 34.222 ms 2024-03-30T14:27:12.331 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:12 smithi031 ceph-mon[24911]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 474 op/s 2024-03-30T14:27:12.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:12 smithi193 ceph-mon[26933]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 474 op/s 2024-03-30T14:27:13.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 118030 12.71 MB/sec execute 118 sec latency 19.878 ms 2024-03-30T14:27:14.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 118709 12.67 MB/sec execute 119 sec latency 23.454 ms 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: pgmap v62: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 769 op/s 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: pgmap v63: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 7.4 MiB/s wr, 700 op/s 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: Failing over to other MGR 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr fail", "who": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr fail", "who": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T14:27:14.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:14 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:27:14.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: pgmap v62: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 769 op/s 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: pgmap v63: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 7.4 MiB/s wr, 700 op/s 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: Upgrade: Need to upgrade myself (mgr.smithi193.ikzlnb) 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: Failing over to other MGR 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 172.21.15.193:0/1346362624' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr fail", "who": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:27:14.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "mgr fail", "who": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:27:14.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T14:27:14.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:14 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:27:15.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 119043 12.62 MB/sec execute 120 sec latency 136.192 ms 2024-03-30T14:27:15.436 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:27:15.436 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd='[{"prefix": "mgr fail", "who": "smithi193.ikzlnb"}]': finished 2024-03-30T14:27:15.436 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: mgrmap e30: smithi031.lkrchi(active, starting, since 0.860409s), standbys: smithi193.ikzlnb 2024-03-30T14:27:15.436 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:27:15.436 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi193.ikzlnb", "id": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:27:15.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:27:15.438 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:27:15.438 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:27:15.438 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:15.438 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:15.438 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:15.438 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:27:15.438 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:15 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14696 ' entity='mgr.smithi193.ikzlnb' cmd='[{"prefix": "mgr fail", "who": "smithi193.ikzlnb"}]': finished 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: mgrmap e30: smithi031.lkrchi(active, starting, since 0.860409s), standbys: smithi193.ikzlnb 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi193.ikzlnb", "id": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:27:15.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:27:15.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:15 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:27:16.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 119430 12.65 MB/sec execute 121 sec latency 54.658 ms 2024-03-30T14:27:16.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:16 smithi031 ceph-mon[24911]: mgrmap e31: smithi031.lkrchi(active, since 1.87308s), standbys: smithi193.ikzlnb 2024-03-30T14:27:16.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:16 smithi031 ceph-mon[24911]: [30/Mar/2024:14:27:15] ENGINE Bus STARTING 2024-03-30T14:27:16.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:16 smithi031 ceph-mon[24911]: [30/Mar/2024:14:27:15] ENGINE Serving on https://172.21.15.31:7150 2024-03-30T14:27:16.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:16 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:16.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:16 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:16.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:16 smithi031 ceph-mon[24911]: [30/Mar/2024:14:27:15] ENGINE Serving on http://172.21.15.31:8765 2024-03-30T14:27:16.466 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:16 smithi031 ceph-mon[24911]: [30/Mar/2024:14:27:15] ENGINE Bus STARTED 2024-03-30T14:27:16.665 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:16 smithi193 ceph-mon[26933]: mgrmap e31: smithi031.lkrchi(active, since 1.87308s), standbys: smithi193.ikzlnb 2024-03-30T14:27:16.665 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:16 smithi193 ceph-mon[26933]: [30/Mar/2024:14:27:15] ENGINE Bus STARTING 2024-03-30T14:27:16.665 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:16 smithi193 ceph-mon[26933]: [30/Mar/2024:14:27:15] ENGINE Serving on https://172.21.15.31:7150 2024-03-30T14:27:16.665 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:16 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:16.665 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:16 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:16.665 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:16 smithi193 ceph-mon[26933]: [30/Mar/2024:14:27:15] ENGINE Serving on http://172.21.15.31:8765 2024-03-30T14:27:16.665 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:16 smithi193 ceph-mon[26933]: [30/Mar/2024:14:27:15] ENGINE Bus STARTED 2024-03-30T14:27:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 119917 12.65 MB/sec execute 122 sec latency 48.013 ms 2024-03-30T14:27:17.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:17 smithi031 ceph-mon[24911]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-30T14:27:17.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:17 smithi031 ceph-mon[24911]: Cluster is now healthy 2024-03-30T14:27:17.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:17 smithi031 ceph-mon[24911]: pgmap v4: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:27:17.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:17 smithi031 ceph-mon[24911]: mgrmap e32: smithi031.lkrchi(active, since 3s), standbys: smithi193.ikzlnb 2024-03-30T14:27:17.531 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:17 smithi193 ceph-mon[26933]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-30T14:27:17.532 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:17 smithi193 ceph-mon[26933]: Cluster is now healthy 2024-03-30T14:27:17.532 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:17 smithi193 ceph-mon[26933]: pgmap v4: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:27:17.532 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:17 smithi193 ceph-mon[26933]: mgrmap e32: smithi031.lkrchi(active, since 3s), standbys: smithi193.ikzlnb 2024-03-30T14:27:18.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 120200 12.62 MB/sec execute 123 sec latency 42.815 ms 2024-03-30T14:27:19.127 INFO:teuthology.orchestra.run.smithi031.stdout: 5 120598 12.66 MB/sec execute 124 sec latency 48.192 ms 2024-03-30T14:27:19.529 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:19 smithi031 ceph-mon[24911]: pgmap v5: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:27:19.529 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:19 smithi031 ceph-mon[24911]: mgrmap e33: smithi031.lkrchi(active, since 4s), standbys: smithi193.ikzlnb 2024-03-30T14:27:19.656 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:19 smithi193 ceph-mon[26933]: pgmap v5: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:27:19.657 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:19 smithi193 ceph-mon[26933]: mgrmap e33: smithi031.lkrchi(active, since 4s), standbys: smithi193.ikzlnb 2024-03-30T14:27:20.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 121068 12.66 MB/sec execute 125 sec latency 45.004 ms 2024-03-30T14:27:21.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 121340 12.60 MB/sec execute 126 sec latency 47.140 ms 2024-03-30T14:27:21.299 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:21 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:21.299 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:21 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:21.299 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:21 smithi031 ceph-mon[24911]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 48 KiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-03-30T14:27:21.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:21 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:21.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:21 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:21.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:21 smithi193 ceph-mon[26933]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 48 KiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-03-30T14:27:22.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 121629 12.57 MB/sec execute 127 sec latency 47.154 ms 2024-03-30T14:27:22.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:22 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:22.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:22 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:22.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:22 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:22.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:22 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 121980 12.48 MB/sec execute 128 sec latency 66.419 ms 2024-03-30T14:27:23.580 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:23 smithi031 ceph-mon[24911]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.6 MiB/s wr, 419 op/s 2024-03-30T14:27:23.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:23 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:27:23.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:23 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:27:23.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:23 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:27:23.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:23 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:27:23.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:23 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:27:23.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:23 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:27:23.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:23 smithi193 ceph-mon[26933]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.6 MiB/s wr, 419 op/s 2024-03-30T14:27:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:23 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:27:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:23 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:27:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:23 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:27:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:23 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:27:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:23 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:27:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:23 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2720137382' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:27:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 122425 12.46 MB/sec execute 129 sec latency 48.727 ms 2024-03-30T14:27:24.538 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:24 smithi031 ceph-mon[24911]: mgrmap e34: smithi031.lkrchi(active, since 9s), standbys: smithi193.ikzlnb 2024-03-30T14:27:24.830 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:24 smithi193 ceph-mon[26933]: mgrmap e34: smithi031.lkrchi(active, since 9s), standbys: smithi193.ikzlnb 2024-03-30T14:27:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 123050 12.53 MB/sec execute 130 sec latency 27.524 ms 2024-03-30T14:27:26.060 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:25 smithi031 ceph-mon[24911]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.6 MiB/s wr, 325 op/s 2024-03-30T14:27:26.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 123662 12.57 MB/sec execute 131 sec latency 45.339 ms 2024-03-30T14:27:26.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:25 smithi193 ceph-mon[26933]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.6 MiB/s wr, 325 op/s 2024-03-30T14:27:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 123971 12.56 MB/sec execute 132 sec latency 42.924 ms 2024-03-30T14:27:27.321 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:27 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:27.322 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:27 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:27.322 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:27 smithi031 ceph-mon[24911]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 546 op/s 2024-03-30T14:27:27.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:27 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:27.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:27 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:27.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:27 smithi193 ceph-mon[26933]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 546 op/s 2024-03-30T14:27:28.126 INFO:teuthology.orchestra.run.smithi031.stdout: 5 124244 12.53 MB/sec execute 133 sec latency 59.269 ms 2024-03-30T14:27:28.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:28 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:28.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:28 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:29.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:28 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:29.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:28 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:29.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 124544 12.49 MB/sec execute 134 sec latency 45.271 ms 2024-03-30T14:27:29.895 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:29 smithi031 ceph-mon[24911]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-03-30T14:27:29.895 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:29 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:29.895 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:29 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:30.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:29 smithi193 ceph-mon[26933]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-03-30T14:27:30.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:29 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:30.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:29 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:30.123 INFO:teuthology.orchestra.run.smithi031.stdout: 5 124824 12.45 MB/sec execute 135 sec latency 67.518 ms 2024-03-30T14:27:31.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 125050 12.41 MB/sec execute 136 sec latency 45.869 ms 2024-03-30T14:27:31.933 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:31 smithi031 ceph-mon[24911]: pgmap v11: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 5.7 MiB/s wr, 529 op/s 2024-03-30T14:27:32.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 125329 12.35 MB/sec execute 137 sec latency 99.494 ms 2024-03-30T14:27:32.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:31 smithi193 ceph-mon[26933]: pgmap v11: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 5.7 MiB/s wr, 529 op/s 2024-03-30T14:27:32.731 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:27:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 125673 12.30 MB/sec execute 138 sec latency 63.301 ms 2024-03-30T14:27:33.736 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:27:33.736 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (105s) 7s ago 10m 16.6M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:27:33.736 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (10m) 7s ago 10m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:27:33.736 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (10m) 13s ago 10m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 unknown 7s ago 10m - - 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (4m) 7s ago 4m 39.7M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (4m) 13s ago 4m 13.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (39s) 7s ago 11m 479M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283 running (2m) 13s ago 10m 109M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 038883e261d6 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (12m) 7s ago 12m 55.3M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (10m) 13s ago 10m 45.9M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (4m) 7s ago 4m 260M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (4m) 13s ago 4m 80.5M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (81s) 7s ago 10m 10.4M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (53s) 13s ago 9m 9047k - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (9m) 7s ago 9m 450M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (8m) 7s ago 8m 396M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (8m) 7s ago 8m 308M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:27:33.737 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (7m) 7s ago 7m 414M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:27:33.738 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (7m) 13s ago 7m 415M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:27:33.738 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (7m) 13s ago 7m 456M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:27:33.738 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (6m) 13s ago 6m 407M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:27:33.738 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (6m) 13s ago 6m 503M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:27:33.738 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (56s) 7s ago 9m 51.3M - 2.33.4 514e6a882f6e c088962ecdba 2024-03-30T14:27:33.941 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:33 smithi031 ceph-mon[24911]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-03-30T14:27:33.941 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:33 smithi031 ceph-mon[24911]: from='client.14840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:34.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 126006 12.25 MB/sec execute 139 sec latency 57.822 ms 2024-03-30T14:27:34.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:33 smithi193 ceph-mon[26933]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-03-30T14:27:34.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:33 smithi193 ceph-mon[26933]: from='client.14840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:34.261 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:27:34.261 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:27:34.261 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:27:34.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:34.262 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:27:34.262 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:27:34.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:34.262 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:27:34.262 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:27:34.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:27:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:27:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:27:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:27:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-30T14:27:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:27:34.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:27:34.264 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "2/23 daemons upgraded", 2024-03-30T14:27:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "", 2024-03-30T14:27:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:27:34.724 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:27:34.961 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:34 smithi031 ceph-mon[24911]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:34.961 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:34 smithi031 ceph-mon[24911]: from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:34.961 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:34 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:34.961 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:34 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:34.961 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:34 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:34.961 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:34 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:27:34.961 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:34 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/862994547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:27:35.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 126285 12.24 MB/sec execute 140 sec latency 51.528 ms 2024-03-30T14:27:35.276 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:27:35.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:34 smithi193 ceph-mon[26933]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:35.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:34 smithi193 ceph-mon[26933]: from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:35.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:34 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:35.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:34 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:35.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:34 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:35.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:34 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:27:35.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:34 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/862994547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:27:35.990 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:27:35.990 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:27:35.990 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.7 MiB/s wr, 464 op/s 2024-03-30T14:27:35.990 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/3890201168' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:35.991 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:35 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.7 MiB/s wr, 464 op/s 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3890201168' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:36.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:36.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:36.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:36.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:36.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:35 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 126600 12.23 MB/sec execute 141 sec latency 59.776 ms 2024-03-30T14:27:37.037 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:36 smithi031 ceph-mon[24911]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:27:37.037 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:36 smithi031 ceph-mon[24911]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:27:37.037 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:36 smithi031 ceph-mon[24911]: pgmap v14: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 695 op/s 2024-03-30T14:27:37.037 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:36 smithi031 ceph-mon[24911]: Reconfiguring prometheus.smithi031 (dependencies changed)... 2024-03-30T14:27:37.037 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:36 smithi031 ceph-mon[24911]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:27:37.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 127081 12.23 MB/sec execute 142 sec latency 59.982 ms 2024-03-30T14:27:37.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:36 smithi193 ceph-mon[26933]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:27:37.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:36 smithi193 ceph-mon[26933]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:27:37.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:36 smithi193 ceph-mon[26933]: pgmap v14: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 695 op/s 2024-03-30T14:27:37.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:36 smithi193 ceph-mon[26933]: Reconfiguring prometheus.smithi031 (dependencies changed)... 2024-03-30T14:27:37.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:36 smithi193 ceph-mon[26933]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:27:38.067 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:37 smithi031 ceph-mon[24911]: Reconfiguring daemon prometheus.smithi031 on smithi031 2024-03-30T14:27:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 127515 12.26 MB/sec execute 143 sec latency 29.748 ms 2024-03-30T14:27:38.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:37 smithi193 ceph-mon[26933]: Reconfiguring daemon prometheus.smithi031 on smithi031 2024-03-30T14:27:39.082 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:38 smithi031 ceph-mon[24911]: pgmap v15: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 422 op/s 2024-03-30T14:27:39.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 127896 12.25 MB/sec execute 144 sec latency 32.753 ms 2024-03-30T14:27:39.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:38 smithi193 ceph-mon[26933]: pgmap v15: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 422 op/s 2024-03-30T14:27:40.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 128211 12.22 MB/sec execute 145 sec latency 52.221 ms 2024-03-30T14:27:41.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:40 smithi193 ceph-mon[26933]: pgmap v16: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 422 op/s 2024-03-30T14:27:41.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 128501 12.21 MB/sec execute 146 sec latency 40.968 ms 2024-03-30T14:27:41.149 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:40 smithi031 ceph-mon[24911]: pgmap v16: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 422 op/s 2024-03-30T14:27:42.124 INFO:teuthology.orchestra.run.smithi031.stdout: 5 128823 12.16 MB/sec execute 147 sec latency 50.631 ms 2024-03-30T14:27:43.125 INFO:teuthology.orchestra.run.smithi031.stdout: 5 129234 12.12 MB/sec execute 148 sec latency 43.855 ms 2024-03-30T14:27:43.178 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:42 smithi031 ceph-mon[24911]: pgmap v17: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.3 MiB/s wr, 632 op/s 2024-03-30T14:27:43.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:42 smithi193 ceph-mon[26933]: pgmap v17: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.3 MiB/s wr, 632 op/s 2024-03-30T14:27:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: 5 129552 12.07 MB/sec execute 149 sec latency 50.298 ms 2024-03-30T14:27:44.971 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:44 smithi031 ceph-mon[24911]: pgmap v18: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 442 op/s 2024-03-30T14:27:44.971 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:44 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:44.971 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:44 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: 5 129888 12.07 MB/sec execute 150 sec latency 48.383 ms 2024-03-30T14:27:45.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:44 smithi193 ceph-mon[26933]: pgmap v18: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 442 op/s 2024-03-30T14:27:45.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:44 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:45.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:44 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: 5 130411 12.09 MB/sec execute 151 sec latency 38.880 ms 2024-03-30T14:27:47.003 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:46 smithi031 ceph-mon[24911]: pgmap v19: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 678 op/s 2024-03-30T14:27:47.125 INFO:teuthology.orchestra.run.smithi031.stdout: 5 130899 12.11 MB/sec execute 152 sec latency 29.277 ms 2024-03-30T14:27:47.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:46 smithi193 ceph-mon[26933]: pgmap v19: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 678 op/s 2024-03-30T14:27:48.125 INFO:teuthology.orchestra.run.smithi031.stdout: 5 131238 12.13 MB/sec execute 153 sec latency 39.136 ms 2024-03-30T14:27:48.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:48 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:48.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:48 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:48.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:48 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:27:48.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:48 smithi031 ceph-mon[24911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:27:48.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:48 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:48.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:48 smithi031 ceph-mon[24911]: pgmap v20: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-03-30T14:27:48.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:48 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:48.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:48 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:48.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:48 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:27:48.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:48 smithi193 ceph-mon[26933]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:27:48.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:48 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:48.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:48 smithi193 ceph-mon[26933]: pgmap v20: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-03-30T14:27:49.130 INFO:teuthology.orchestra.run.smithi031.stdout: 5 131587 12.10 MB/sec execute 154 sec latency 55.990 ms 2024-03-30T14:27:49.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:49 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:27:49.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:49 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:27:49.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:49 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:49.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:49 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:27:49.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:49 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:27:49.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:49 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:50.130 INFO:teuthology.orchestra.run.smithi031.stdout: 5 131840 12.07 MB/sec execute 155 sec latency 54.452 ms 2024-03-30T14:27:50.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:50 smithi193 ceph-mon[26933]: Upgrade: Updating mgr.smithi193.ikzlnb 2024-03-30T14:27:50.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:50 smithi193 ceph-mon[26933]: Deploying daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:27:50.999 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:50 smithi193 ceph-mon[26933]: pgmap v21: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-03-30T14:27:51.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:50 smithi031 ceph-mon[24911]: Upgrade: Updating mgr.smithi193.ikzlnb 2024-03-30T14:27:51.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:50 smithi031 ceph-mon[24911]: Deploying daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:27:51.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:50 smithi031 ceph-mon[24911]: pgmap v21: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-03-30T14:27:51.130 INFO:teuthology.orchestra.run.smithi031.stdout: 5 132083 12.04 MB/sec execute 156 sec latency 52.372 ms 2024-03-30T14:27:52.130 INFO:teuthology.orchestra.run.smithi031.stdout: 5 132399 12.00 MB/sec execute 157 sec latency 60.673 ms 2024-03-30T14:27:53.040 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:52 smithi193 ceph-mon[26933]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.0 MiB/s wr, 673 op/s 2024-03-30T14:27:53.094 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:52 smithi031 ceph-mon[24911]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.0 MiB/s wr, 673 op/s 2024-03-30T14:27:53.130 INFO:teuthology.orchestra.run.smithi031.stdout: 5 132840 11.96 MB/sec execute 158 sec latency 43.439 ms 2024-03-30T14:27:54.130 INFO:teuthology.orchestra.run.smithi031.stdout: 5 133165 11.92 MB/sec execute 159 sec latency 42.576 ms 2024-03-30T14:27:55.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 133451 11.91 MB/sec execute 160 sec latency 52.797 ms 2024-03-30T14:27:55.137 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:54 smithi031 ceph-mon[24911]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.4 MiB/s wr, 446 op/s 2024-03-30T14:27:55.138 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:54 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:55.138 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:54 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:55.138 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:54 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:55.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:54 smithi193 ceph-mon[26933]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.4 MiB/s wr, 446 op/s 2024-03-30T14:27:55.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:54 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:55.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:54 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:55.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:54 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:56.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 133798 11.91 MB/sec execute 161 sec latency 51.596 ms 2024-03-30T14:27:56.928 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:56 smithi031 ceph-mon[24911]: pgmap v24: 97 pgs: 97 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 660 op/s 2024-03-30T14:27:57.054 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:56 smithi193 ceph-mon[26933]: pgmap v24: 97 pgs: 97 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 660 op/s 2024-03-30T14:27:57.130 INFO:teuthology.orchestra.run.smithi031.stdout: 5 134187 11.91 MB/sec execute 162 sec latency 53.251 ms 2024-03-30T14:27:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 134496 11.90 MB/sec execute 163 sec latency 54.561 ms 2024-03-30T14:27:59.032 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:58 smithi031 ceph-mon[24911]: pgmap v25: 97 pgs: 97 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-03-30T14:27:59.032 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:58 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:59.032 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:58 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:59.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 134713 11.88 MB/sec execute 164 sec latency 49.217 ms 2024-03-30T14:27:59.324 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:58 smithi193 ceph-mon[26933]: pgmap v25: 97 pgs: 97 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-03-30T14:27:59.324 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:58 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:27:59.324 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:58 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:00.041 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:59 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:00.041 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:59 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:00.041 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:27:59 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:00.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 135077 11.87 MB/sec execute 165 sec latency 54.607 ms 2024-03-30T14:28:00.214 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:59 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:00.214 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:59 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:00.214 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:27:59 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:01.061 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:00 smithi031 ceph-mon[24911]: pgmap v26: 97 pgs: 97 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-03-30T14:28:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 135402 11.85 MB/sec execute 166 sec latency 55.096 ms 2024-03-30T14:28:01.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:00 smithi193 ceph-mon[26933]: pgmap v26: 97 pgs: 97 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-03-30T14:28:02.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 135688 11.82 MB/sec execute 167 sec latency 61.459 ms 2024-03-30T14:28:03.089 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:02 smithi031 ceph-mon[24911]: pgmap v27: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.6 MiB/s wr, 622 op/s 2024-03-30T14:28:03.089 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:02 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:28:03.090 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:02 smithi031 ceph-mon[24911]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:28:03.090 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:02 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:28:03.090 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:02 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:28:03.090 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:02 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:28:03.090 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:02 smithi031 ceph-mon[24911]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:28:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 136259 11.81 MB/sec execute 168 sec latency 33.779 ms 2024-03-30T14:28:03.246 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:02 smithi193 ceph-mon[26933]: pgmap v27: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.6 MiB/s wr, 622 op/s 2024-03-30T14:28:03.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:02 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:28:03.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:02 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:28:03.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:02 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:28:03.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:02 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:28:03.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:02 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:28:03.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:02 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/2022747518' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:28:04.109 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: mgrmap e35: smithi031.lkrchi(active, since 49s), standbys: smithi193.ikzlnb 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi031.lkrchi"}]: dispatch 2024-03-30T14:28:04.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi031.lkrchi"}]': finished 2024-03-30T14:28:04.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi193.ikzlnb"}]: dispatch 2024-03-30T14:28:04.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi193.ikzlnb"}]': finished 2024-03-30T14:28:04.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:03 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T14:28:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 136805 11.81 MB/sec execute 169 sec latency 28.194 ms 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: mgrmap e35: smithi031.lkrchi(active, since 49s), standbys: smithi193.ikzlnb 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi031.lkrchi"}]: dispatch 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi031.lkrchi"}]': finished 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi193.ikzlnb"}]: dispatch 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi193.ikzlnb"}]': finished 2024-03-30T14:28:04.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:03 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T14:28:05.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:04 smithi031 ceph-mon[24911]: pgmap v28: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 397 op/s 2024-03-30T14:28:05.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:04 smithi031 ceph-mon[24911]: pgmap v29: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-03-30T14:28:05.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:04 smithi031 ceph-mon[24911]: Upgrade: Setting container_image for all mgr 2024-03-30T14:28:05.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 137361 11.85 MB/sec execute 170 sec latency 32.065 ms 2024-03-30T14:28:05.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:04 smithi193 ceph-mon[26933]: pgmap v28: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 397 op/s 2024-03-30T14:28:05.979 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:04 smithi193 ceph-mon[26933]: pgmap v29: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-03-30T14:28:05.979 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:04 smithi193 ceph-mon[26933]: Upgrade: Setting container_image for all mgr 2024-03-30T14:28:05.980 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:28:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 137923 11.88 MB/sec execute 171 sec latency 50.908 ms 2024-03-30T14:28:06.946 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (2m) 3s ago 11m 16.6M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (11m) 3s ago 11m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (10m) 8s ago 10m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 unknown 3s ago 11m - - dad864ee21e9 c50ed69b60d3 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (5m) 3s ago 5m 39.5M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (5m) 8s ago 5m 13.8M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (72s) 3s ago 12m 494M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (13s) 8s ago 10m 66.4M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (12m) 3s ago 12m 56.8M 2048M 17.2.0 e1d6a67b021e 1cdb8d9d80b4 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (10m) 8s ago 10m 45.4M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (5m) 3s ago 5m 265M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (5m) 8s ago 4m 82.1M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (115s) 3s ago 11m 10.3M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (86s) 8s ago 10m 13.5M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (9m) 3s ago 9m 485M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (9m) 3s ago 9m 410M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (8m) 3s ago 8m 355M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (8m) 3s ago 8m 443M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (8m) 8s ago 8m 478M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (7m) 8s ago 7m 524M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (7m) 8s ago 7m 440M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (6m) 8s ago 6m 548M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (20s) 3s ago 10m 48.1M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:28:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 138367 11.90 MB/sec execute 172 sec latency 32.870 ms 2024-03-30T14:28:07.143 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:06 smithi031 ceph-mon[24911]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-03-30T14:28:07.143 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:06 smithi031 ceph-mon[24911]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:07.143 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:06 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:07.143 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:06 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:28:07.143 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:06 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:28:07.144 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:06 smithi031 ceph-mon[24911]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:07.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:06 smithi193 ceph-mon[26933]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-03-30T14:28:07.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:06 smithi193 ceph-mon[26933]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:07.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:06 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:07.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:06 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:28:07.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:06 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:28:07.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:06 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:07.674 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:28:07.674 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:28:07.674 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:28:07.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:07.674 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:28:07.675 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:28:08.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 138882 11.90 MB/sec execute 173 sec latency 31.559 ms 2024-03-30T14:28:08.151 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:07 smithi031 ceph-mon[24911]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:08.151 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:07 smithi031 ceph-mon[24911]: Upgrade: Updating mon.smithi031 2024-03-30T14:28:08.152 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:07 smithi031 ceph-mon[24911]: Deploying daemon mon.smithi031 on smithi031 2024-03-30T14:28:08.152 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:07 smithi031 ceph-mon[24911]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:08.152 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:07 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/511588614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "2/23 daemons upgraded", 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading mon daemons", 2024-03-30T14:28:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:28:08.268 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:28:08.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:07 smithi193 ceph-mon[26933]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:08.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:07 smithi193 ceph-mon[26933]: Upgrade: Updating mon.smithi031 2024-03-30T14:28:08.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:07 smithi193 ceph-mon[26933]: Deploying daemon mon.smithi031 on smithi031 2024-03-30T14:28:08.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:07 smithi193 ceph-mon[26933]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:08.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:07 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/511588614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:08.996 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:08.996 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:08.996 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in unknown state 2024-03-30T14:28:09.035 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:08 smithi031 ceph-mon[24911]: pgmap v31: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-03-30T14:28:09.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 139315 11.90 MB/sec execute 174 sec latency 31.032 ms 2024-03-30T14:28:09.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:08 smithi193 ceph-mon[26933]: pgmap v31: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-03-30T14:28:10.131 INFO:teuthology.orchestra.run.smithi031.stdout: 5 139884 11.89 MB/sec execute 175 sec latency 41.367 ms 2024-03-30T14:28:10.162 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:09 smithi031 ceph-mon[24911]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:10.162 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:09 smithi031 ceph-mon[24911]: from='client.? 172.21.15.31:0/2259124707' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:28:10.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:09 smithi193 ceph-mon[26933]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:10.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:09 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/2259124707' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:28:10.941 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:10 smithi031 systemd[1]: Stopping Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:28:11.132 INFO:teuthology.orchestra.run.smithi031.stdout: 5 140345 11.88 MB/sec execute 176 sec latency 54.826 ms 2024-03-30T14:28:11.191 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:10 smithi031 ceph-mon[24911]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-03-30T14:28:11.192 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:11 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[24907]: 2024-03-30T14:28:11.133+0000 7f607130a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi031 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:28:11.192 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:11 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[24907]: 2024-03-30T14:28:11.133+0000 7f607130a700 -1 mon.smithi031@0(leader) e2 *** Got Signal Terminated *** 2024-03-30T14:28:11.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:10 smithi193 ceph-mon[26933]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-03-30T14:28:11.443 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:11 smithi031 podman[67286]: 2024-03-30 14:28:11.421178503 +0000 UTC m=+0.428084318 container died 1cdb8d9d80b4568668e0861d6ffe9c3f6331b08a1f2c88439a91c9e950436b64 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, version=8, GIT_BRANCH=HEAD, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, ceph=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, architecture=x86_64, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8) 2024-03-30T14:28:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: 5 140876 11.91 MB/sec execute 177 sec latency 43.961 ms 2024-03-30T14:28:12.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:12 smithi031 podman[67286]: 2024-03-30 14:28:12.069552163 +0000 UTC m=+1.076457970 container cleanup 1cdb8d9d80b4568668e0861d6ffe9c3f6331b08a1f2c88439a91c9e950436b64 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.component=centos-stream-container, name=centos-stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8) 2024-03-30T14:28:12.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:12 smithi031 bash[67286]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031 2024-03-30T14:28:12.715 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:12 smithi031 podman[67311]: 2024-03-30 14:28:12.548335167 +0000 UTC m=+1.126000911 container remove 1cdb8d9d80b4568668e0861d6ffe9c3f6331b08a1f2c88439a91c9e950436b64 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.openshift.expose-services=, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-type=git, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, release=754, name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-30T14:28:12.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:12 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service: Deactivated successfully. 2024-03-30T14:28:12.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:12 smithi031 systemd[1]: Stopped Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:28:12.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:12 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service: Consumed 16.700s CPU time. 2024-03-30T14:28:13.132 INFO:teuthology.orchestra.run.smithi031.stdout: 5 141430 11.94 MB/sec execute 178 sec latency 37.819 ms 2024-03-30T14:28:13.217 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:13 smithi031 systemd[1]: Starting Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:28:13.468 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:13 smithi031 podman[67439]: 2024-03-30 14:28:13.249136695 +0000 UTC m=+0.039719592 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:28:13.719 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:13 smithi031 podman[67439]: 2024-03-30 14:28:13.624324877 +0000 UTC m=+0.414907759 container create 6ab49835e68fc948a4688d4b0102831d8dbe5751f91365a124236b4c49fc44f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-30T14:28:14.132 INFO:teuthology.orchestra.run.smithi031.stdout: 5 141891 11.96 MB/sec execute 179 sec latency 30.901 ms 2024-03-30T14:28:14.494 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 podman[67439]: 2024-03-30 14:28:14.378499039 +0000 UTC m=+1.169081927 container init 6ab49835e68fc948a4688d4b0102831d8dbe5751f91365a124236b4c49fc44f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-30T14:28:14.494 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 podman[67439]: 2024-03-30 14:28:14.389960917 +0000 UTC m=+1.180543800 container start 6ab49835e68fc948a4688d4b0102831d8dbe5751f91365a124236b4c49fc44f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-30T14:28:14.494 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 2 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: pidfile_write: ignore empty --pid-file 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: load: jerasure load: lrc 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: RocksDB version: 7.9.2 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Git sha 0 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: DB SUMMARY 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: DB Session ID: FB9OLJG7U2L4IE9ALP3J 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: CURRENT file: CURRENT 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi031/store.db dir, Total Num: 1, files: 000030.sst 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi031/store.db: 000028.log size: 2742590 ; 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.error_if_exists: 0 2024-03-30T14:28:14.495 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.create_if_missing: 0 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.paranoid_checks: 1 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.env: 0x55ebb568ec60 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.info_log: 0x55ebb61799a0 2024-03-30T14:28:14.496 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.statistics: (nil) 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.use_fsync: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_log_file_size: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.allow_fallocate: 1 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.use_direct_reads: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T14:28:14.497 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.db_log_dir: 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.wal_dir: 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.write_buffer_manager: 0x55ebb614c1e0 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T14:28:14.498 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.unordered_write: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.row_cache: None 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.wal_filter: None 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.two_write_queues: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.wal_compression: 0 2024-03-30T14:28:14.499 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.atomic_flush: 0 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.log_readahead_size: 0 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T14:28:14.500 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_background_jobs: 2 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_background_compactions: -1 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_subcompactions: 1 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_open_files: -1 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T14:28:14.501 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_background_flushes: -1 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Compression algorithms supported: 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kZSTD supported: 0 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kXpressCompression supported: 0 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kZlibCompression supported: 1 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kSnappyCompression supported: 1 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kLZ4Compression supported: 1 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: kBZip2Compression supported: 0 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi031/store.db/MANIFEST-000015 2024-03-30T14:28:14.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.merge_operator: 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_filter: None 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ebb6179aa0) 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T14:28:14.503 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_type: 0 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: data_block_index_type: 0 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_shortening: 1 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: checksum: 4 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: no_block_cache: 0 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache: 0x55ebb617d090 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_options: 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: capacity : 536870912 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: num_shard_bits : 4 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: strict_capacity_limit : 0 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-03-30T14:28:14.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_compressed: (nil) 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: persistent_cache: (nil) 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_size: 4096 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_size_deviation: 10 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_restart_interval: 16 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_block_restart_interval: 1 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: metadata_block_size: 4096 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: partition_filters: 0 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: use_delta_encoding: 1 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: filter_policy: bloomfilter 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: whole_key_filtering: 1 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: verify_compression: 0 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: format_version: 5 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: enable_index_compression: 1 2024-03-30T14:28:14.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_align: 0 2024-03-30T14:28:14.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: max_auto_readahead_size: 262144 2024-03-30T14:28:14.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: prepopulate_block_cache: 0 2024-03-30T14:28:14.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: initial_auto_readahead_size: 8192 2024-03-30T14:28:14.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression: NoCompression 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.num_levels: 7 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T14:28:14.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:28:14.508 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T14:28:14.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T14:28:14.510 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.table_properties_collectors: 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.inplace_update_support: 0 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.bloom_locality: 0 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.max_successive_merges: 0 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T14:28:14.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.ttl: 2592000 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.enable_blob_files: false 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.min_blob_size: 0 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T14:28:14.512 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi031/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11855, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: df62afca-be6c-4f49-8eb2-a21bae5505ff 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808894433894, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808894444137, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2541712, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11842, "largest_seqno": 12978, "table_properties": {"data_size": 2536133, "index_size": 3344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1221, "raw_key_size": 11214, "raw_average_key_size": 23, "raw_value_size": 2525556, "raw_average_value_size": 5196, "num_data_blocks": 152, "num_entries": 486, "num_filter_entries": 486, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711808894, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "df62afca-be6c-4f49-8eb2-a21bae5505ff", "db_session_id": "FB9OLJG7U2L4IE9ALP3J", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808894444295, "job": 1, "event": "recovery_finished"} 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi031/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T14:28:14.513 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ebb626a000 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: DB pointer 0x55ebb6258000 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** DB Stats ** 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:28:14.514 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: L0 1/0 2.42 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: L6 1/0 7.02 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Sum 2/0 9.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 535.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-30T14:28:14.515 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative compaction: 0.00 GB write, 160.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval compaction: 0.00 GB write, 160.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Block cache BinnedLRUCache@0x55ebb617d090#2 capacity: 512.00 MB usage: 51.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Block cache entry stats(count,size,portion): DataBlock(3,20.73 KB,0.00395477%) FilterBlock(2,10.31 KB,0.00196695%) IndexBlock(2,20.16 KB,0.0038445%) Misc(1,0.00 KB,0%) 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: starting mon.smithi031 rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi031 fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???) e2 preinit fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???).mds e7 new map 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???).mds e7 print_map 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: e7 2024-03-30T14:28:14.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: legacy client fscid: 1 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Filesystem 'foofs' (1) 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: fs_name foofs 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: epoch 5 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: created 2024-03-30T14:22:35.894607+0000 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: modified 2024-03-30T14:22:41.925357+0000 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: tableserver 0 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: root 0 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: session_timeout 60 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: session_autoclose 300 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: max_file_size 1099511627776 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: max_xattr_size 65536 2024-03-30T14:28:14.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: required_client_features {} 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: last_failure 0 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: last_failure_osd_epoch 0 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: max_mds 1 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: in 0 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: up {0=14498} 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: failed 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: damaged 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: stopped 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: data_pools [3] 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: metadata_pool 2 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: inline_data disabled 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: balancer 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: bal_rank_mask -1 2024-03-30T14:28:14.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: standby_count_wanted 1 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: qdb_cluster leader: 0 members: 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: [mds.foofs.smithi031.xsynqy{0:14498} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.31:6834/669631789,v1:172.21.15.31:6835/669631789] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Standby daemons: 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: [mds.foofs.smithi193.shgxxs{-1:24301} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-30T14:28:14.519 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 ceph-mon[67456]: mon.smithi031@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-03-30T14:28:14.995 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 bash[67439]: 6ab49835e68fc948a4688d4b0102831d8dbe5751f91365a124236b4c49fc44f8 2024-03-30T14:28:14.995 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:14 smithi031 systemd[1]: Started Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:28:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 142305 11.96 MB/sec execute 180 sec latency 32.515 ms 2024-03-30T14:28:16.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 142718 11.96 MB/sec execute 181 sec latency 114.716 ms 2024-03-30T14:28:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 142871 11.90 MB/sec execute 182 sec latency 1114.765 ms 2024-03-30T14:28:17.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:28:17.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:28:17.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: pgmap v33: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 711 op/s 2024-03-30T14:28:17.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: pgmap v34: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 711 op/s 2024-03-30T14:28:17.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:17.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: pgmap v35: 97 pgs: 97 active+clean; 345 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 882 op/s 2024-03-30T14:28:17.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: mon.smithi031 calling monitor election 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: mon.smithi031 is new leader, mons smithi031,smithi193 in quorum (ranks 0,1) 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: monmap epoch 2 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: last_changed 2024-03-30T14:17:31.578833+0000 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: created 2024-03-30T14:15:16.778426+0000 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: min_mon_release 17 (quincy) 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: election_strategy: 1 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.smithi031 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.smithi193 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: mgrmap e35: smithi031.lkrchi(active, since 63s), standbys: smithi193.ikzlnb 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:17.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:17.547 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: daemon grafana.smithi031 on smithi031 is in unknown state 2024-03-30T14:28:17.547 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: from='mgr.14784 ' entity='' 2024-03-30T14:28:17.547 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:17 smithi031 ceph-mon[67456]: mgrmap e36: smithi031.lkrchi(active, since 63s), standbys: smithi193.ikzlnb 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: pgmap v33: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 711 op/s 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: pgmap v34: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 711 op/s 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: from='mgr.14784 172.21.15.31:0/2582063324' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: pgmap v35: 97 pgs: 97 active+clean; 345 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 882 op/s 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: mon.smithi031 calling monitor election 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: mon.smithi031 is new leader, mons smithi031,smithi193 in quorum (ranks 0,1) 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: monmap epoch 2 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: last_changed 2024-03-30T14:17:31.578833+0000 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: created 2024-03-30T14:15:16.778426+0000 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: min_mon_release 17 (quincy) 2024-03-30T14:28:17.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: election_strategy: 1 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.smithi031 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.smithi193 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: mgrmap e35: smithi031.lkrchi(active, since 63s), standbys: smithi193.ikzlnb 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: daemon grafana.smithi031 on smithi031 is in unknown state 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: from='mgr.14784 ' entity='' 2024-03-30T14:28:17.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:17 smithi193 ceph-mon[26933]: mgrmap e36: smithi031.lkrchi(active, since 63s), standbys: smithi193.ikzlnb 2024-03-30T14:28:18.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 143114 11.87 MB/sec execute 183 sec latency 1451.754 ms 2024-03-30T14:28:19.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 143566 11.86 MB/sec execute 184 sec latency 54.855 ms 2024-03-30T14:28:19.492 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:19 smithi031 ceph-mon[67456]: mgrmap e37: smithi031.lkrchi(active, since 64s), standbys: smithi193.ikzlnb 2024-03-30T14:28:19.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:19 smithi193 ceph-mon[26933]: mgrmap e37: smithi031.lkrchi(active, since 64s), standbys: smithi193.ikzlnb 2024-03-30T14:28:20.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 143921 11.83 MB/sec execute 185 sec latency 38.290 ms 2024-03-30T14:28:21.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 144152 11.81 MB/sec execute 186 sec latency 58.952 ms 2024-03-30T14:28:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 144451 11.80 MB/sec execute 187 sec latency 55.132 ms 2024-03-30T14:28:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 144783 11.78 MB/sec execute 188 sec latency 56.508 ms 2024-03-30T14:28:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 145302 11.80 MB/sec execute 189 sec latency 25.100 ms 2024-03-30T14:28:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 145693 11.81 MB/sec execute 190 sec latency 25.386 ms 2024-03-30T14:28:26.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:25 smithi031 ceph-mon[67456]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:28:26.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:25 smithi031 ceph-mon[67456]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:28:26.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:25 smithi031 ceph-mon[67456]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:28:26.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:25 smithi031 ceph-mon[67456]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:28:26.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:25 smithi031 ceph-mon[67456]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:28:26.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:25 smithi031 ceph-mon[67456]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:28:26.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:25 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb restarted 2024-03-30T14:28:26.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:25 smithi193 ceph-mon[26933]: Standby manager daemon smithi193.ikzlnb started 2024-03-30T14:28:26.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:25 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/crt"}]: dispatch 2024-03-30T14:28:26.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:25 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:28:26.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:25 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.ikzlnb/key"}]: dispatch 2024-03-30T14:28:26.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:25 smithi193 ceph-mon[26933]: from='mgr.? 172.21.15.193:0/3236569005' entity='mgr.smithi193.ikzlnb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:28:26.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 146141 11.80 MB/sec execute 191 sec latency 46.940 ms 2024-03-30T14:28:27.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:26 smithi031 ceph-mon[67456]: mgrmap e38: smithi031.lkrchi(active, since 72s), standbys: smithi193.ikzlnb 2024-03-30T14:28:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 146506 11.79 MB/sec execute 192 sec latency 38.666 ms 2024-03-30T14:28:27.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:26 smithi193 ceph-mon[26933]: mgrmap e38: smithi031.lkrchi(active, since 72s), standbys: smithi193.ikzlnb 2024-03-30T14:28:27.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193[26927]: 2024-03-30T14:28:27.453+0000 7f96e75cc700 -1 mon.smithi193@1(peon).osd e73 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-30T14:28:28.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: Active manager daemon smithi031.lkrchi restarted 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: Activating manager daemon smithi031.lkrchi 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: failed to encode map e73 with expected crc 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: mgrmap e39: smithi031.lkrchi(active, starting, since 0.00876298s), standbys: smithi193.ikzlnb 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi193.ikzlnb", "id": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:28:28.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:28:28.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:27 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:28:28.109 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: Active manager daemon smithi031.lkrchi restarted 2024-03-30T14:28:28.109 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: Activating manager daemon smithi031.lkrchi 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: failed to encode map e73 with expected crc 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: mgrmap e39: smithi031.lkrchi(active, starting, since 0.00876298s), standbys: smithi193.ikzlnb 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi031.lkrchi", "id": "smithi031.lkrchi"}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr metadata", "who": "smithi193.ikzlnb", "id": "smithi193.ikzlnb"}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:28:28.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: Manager daemon smithi031.lkrchi is now available 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:28:28.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi031.lkrchi/trash_purge_schedule"}]: dispatch 2024-03-30T14:28:28.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 146768 11.76 MB/sec execute 193 sec latency 40.889 ms 2024-03-30T14:28:29.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:28 smithi031 ceph-mon[67456]: failed to encode map e73 with expected crc 2024-03-30T14:28:29.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:28 smithi031 ceph-mon[67456]: [30/Mar/2024:14:28:28] ENGINE Bus STARTING 2024-03-30T14:28:29.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:28 smithi031 ceph-mon[67456]: [30/Mar/2024:14:28:28] ENGINE Serving on https://172.21.15.31:7150 2024-03-30T14:28:29.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:28 smithi031 ceph-mon[67456]: mgrmap e40: smithi031.lkrchi(active, since 1.01352s), standbys: smithi193.ikzlnb 2024-03-30T14:28:29.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:28 smithi031 ceph-mon[67456]: pgmap v3: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:29.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:28 smithi031 ceph-mon[67456]: [30/Mar/2024:14:28:28] ENGINE Serving on http://172.21.15.31:8765 2024-03-30T14:28:29.123 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:28 smithi031 ceph-mon[67456]: [30/Mar/2024:14:28:28] ENGINE Bus STARTED 2024-03-30T14:28:29.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 147208 11.75 MB/sec execute 194 sec latency 41.120 ms 2024-03-30T14:28:29.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:28 smithi193 ceph-mon[26933]: failed to encode map e73 with expected crc 2024-03-30T14:28:29.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:28 smithi193 ceph-mon[26933]: [30/Mar/2024:14:28:28] ENGINE Bus STARTING 2024-03-30T14:28:29.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:28 smithi193 ceph-mon[26933]: [30/Mar/2024:14:28:28] ENGINE Serving on https://172.21.15.31:7150 2024-03-30T14:28:29.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:28 smithi193 ceph-mon[26933]: mgrmap e40: smithi031.lkrchi(active, since 1.01352s), standbys: smithi193.ikzlnb 2024-03-30T14:28:29.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:28 smithi193 ceph-mon[26933]: pgmap v3: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:29.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:28 smithi193 ceph-mon[26933]: [30/Mar/2024:14:28:28] ENGINE Serving on http://172.21.15.31:8765 2024-03-30T14:28:29.247 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:28 smithi193 ceph-mon[26933]: [30/Mar/2024:14:28:28] ENGINE Bus STARTED 2024-03-30T14:28:29.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:29 smithi031 ceph-mon[67456]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-30T14:28:29.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:29 smithi031 ceph-mon[67456]: Cluster is now healthy 2024-03-30T14:28:29.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:29 smithi031 ceph-mon[67456]: pgmap v4: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 147568 11.73 MB/sec execute 195 sec latency 48.245 ms 2024-03-30T14:28:30.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:29 smithi193 ceph-mon[26933]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-30T14:28:30.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:29 smithi193 ceph-mon[26933]: Cluster is now healthy 2024-03-30T14:28:30.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:29 smithi193 ceph-mon[26933]: pgmap v4: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:30.904 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:30 smithi031 ceph-mon[67456]: mgrmap e41: smithi031.lkrchi(active, since 2s), standbys: smithi193.ikzlnb 2024-03-30T14:28:31.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:30 smithi193 ceph-mon[26933]: mgrmap e41: smithi031.lkrchi(active, since 2s), standbys: smithi193.ikzlnb 2024-03-30T14:28:31.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 147851 11.71 MB/sec execute 196 sec latency 45.096 ms 2024-03-30T14:28:31.918 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:31 smithi031 ceph-mon[67456]: pgmap v5: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:32.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:31 smithi193 ceph-mon[26933]: pgmap v5: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:32.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 148292 11.73 MB/sec execute 197 sec latency 43.575 ms 2024-03-30T14:28:32.929 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:32 smithi031 ceph-mon[67456]: mgrmap e42: smithi031.lkrchi(active, since 4s), standbys: smithi193.ikzlnb 2024-03-30T14:28:32.930 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:32.930 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:33.133 INFO:teuthology.orchestra.run.smithi031.stdout: 5 148644 11.71 MB/sec execute 198 sec latency 37.574 ms 2024-03-30T14:28:33.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:32 smithi193 ceph-mon[26933]: mgrmap e42: smithi031.lkrchi(active, since 4s), standbys: smithi193.ikzlnb 2024-03-30T14:28:33.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:32 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:33.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:32 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:34.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 148932 11.70 MB/sec execute 199 sec latency 40.066 ms 2024-03-30T14:28:34.718 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:34 smithi031 ceph-mon[67456]: pgmap v6: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:34.719 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:34.719 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:34.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:34 smithi193 ceph-mon[26933]: pgmap v6: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-30T14:28:34.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:34 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:34.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:34 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:35.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 149217 11.69 MB/sec execute 200 sec latency 48.993 ms 2024-03-30T14:28:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 149710 11.71 MB/sec execute 201 sec latency 37.635 ms 2024-03-30T14:28:36.764 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:36 smithi031 ceph-mon[67456]: pgmap v7: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-03-30T14:28:36.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:36 smithi193 ceph-mon[26933]: pgmap v7: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-03-30T14:28:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 150159 11.68 MB/sec execute 202 sec latency 31.766 ms 2024-03-30T14:28:38.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 150640 11.69 MB/sec execute 203 sec latency 33.372 ms 2024-03-30T14:28:38.271 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.522 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.522 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.522 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.522 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:38 smithi031 ceph-mon[67456]: pgmap v8: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-30T14:28:38.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:38 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:38 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:38 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:38 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:38.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:38 smithi193 ceph-mon[26933]: pgmap v8: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-30T14:28:39.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 151086 11.68 MB/sec execute 204 sec latency 51.998 ms 2024-03-30T14:28:39.657 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:28:40.050 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:40 smithi031 ceph-mon[67456]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-03-30T14:28:40.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 151358 11.66 MB/sec execute 205 sec latency 42.613 ms 2024-03-30T14:28:40.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:40.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:40.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:40.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:40.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-30T14:28:40.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T14:28:40.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:28:40.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:28:40.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:40 smithi193 ceph-mon[26933]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (2m) 3s ago 11m 16.6M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (11m) 3s ago 11m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (11m) 8s ago 11m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 3s ago 11m - - 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (6m) 3s ago 6m 40.0M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (5m) 8s ago 5m 14.1M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (106s) 3s ago 13m 477M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (47s) 8s ago 11m 447M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:28:40.852 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (27s) 3s ago 13m 31.7M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (11m) 8s ago 11m 51.2M 2048M 17.2.0 e1d6a67b021e 12e88782f847 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (5m) 3s ago 5m 272M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (5m) 8s ago 5m 82.2M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (2m) 3s ago 11m 12.0M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (2m) 8s ago 11m 16.0M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (10m) 3s ago 10m 523M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (9m) 3s ago 9m 453M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (9m) 3s ago 9m 389M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (9m) 3s ago 9m 487M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (8m) 8s ago 8m 493M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (8m) 8s ago 8m 316M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (7m) 8s ago 7m 498M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (7m) 8s ago 7m 585M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:28:40.853 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (54s) 3s ago 10m 55.4M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:28:41.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 151768 11.67 MB/sec execute 206 sec latency 101.622 ms 2024-03-30T14:28:41.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:28:41.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='client.24645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:41.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:28:41.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:28:41.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:28:41.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:41.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:41.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T14:28:41.508 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:28:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:28:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:28:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:28:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 3 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:28:41.509 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:28:41.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='client.24645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: Updating smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: Updating smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:41 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T14:28:42.124 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:28:42.124 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:28:42.124 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:28:42.124 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:28:42.124 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:28:42.125 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:28:42.125 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:28:42.125 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "3/23 daemons upgraded", 2024-03-30T14:28:42.125 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading mon daemons", 2024-03-30T14:28:42.125 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:28:42.125 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:28:42.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 152208 11.67 MB/sec execute 207 sec latency 39.128 ms 2024-03-30T14:28:42.330 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:42 smithi031 ceph-mon[67456]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-03-30T14:28:42.330 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:42 smithi031 ceph-mon[67456]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:42.330 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:42 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/3442851508' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:42.330 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:42 smithi031 ceph-mon[67456]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:28:42.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:42 smithi193 ceph-mon[26933]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-03-30T14:28:42.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:42 smithi193 ceph-mon[26933]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:42.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:42 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/3442851508' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:28:42.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:42 smithi193 ceph-mon[26933]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:28:42.773 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:42.774 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:42.774 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:28:43.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 152526 11.67 MB/sec execute 208 sec latency 36.225 ms 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: Upgrade: Updating mon.smithi193 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: Deploying daemon mon.smithi193 on smithi193 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:43.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:43 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/4174945802' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:28:43.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: Upgrade: Updating mon.smithi193 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: Deploying daemon mon.smithi193 on smithi193 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:43.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:43 smithi193 ceph-mon[26933]: from='client.? 172.21.15.31:0/4174945802' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:28:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 153067 11.69 MB/sec execute 209 sec latency 40.562 ms 2024-03-30T14:28:44.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:44 smithi193 ceph-mon[26933]: pgmap v11: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-03-30T14:28:44.609 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:44 smithi031 ceph-mon[67456]: pgmap v11: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-03-30T14:28:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 153541 11.68 MB/sec execute 210 sec latency 40.520 ms 2024-03-30T14:28:45.296 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:44 smithi193 systemd[1]: Stopping Ceph mon.smithi193 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:28:45.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193[26927]: 2024-03-30T14:28:45.298+0000 7f96ecdd7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi193 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:28:45.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193[26927]: 2024-03-30T14:28:45.298+0000 7f96ecdd7700 -1 mon.smithi193@1(peon) e2 *** Got Signal Terminated *** 2024-03-30T14:28:45.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:45 smithi193 podman[52816]: 2024-03-30 14:28:45.36106386 +0000 UTC m=+0.183584829 container died 12e88782f8473e9b4e0223fa1a608000cf363d0373b450fd99f0fd6c6b08189d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vcs-type=git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_BRANCH=HEAD) 2024-03-30T14:28:46.036 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:45 smithi193 podman[52816]: 2024-03-30 14:28:45.765472289 +0000 UTC m=+0.587993264 container cleanup 12e88782f8473e9b4e0223fa1a608000cf363d0373b450fd99f0fd6c6b08189d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, ceph=True, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, architecture=x86_64, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-30T14:28:46.036 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:45 smithi193 bash[52816]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193 2024-03-30T14:28:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: 5 153853 11.66 MB/sec execute 211 sec latency 37.255 ms 2024-03-30T14:28:46.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 podman[52830]: 2024-03-30 14:28:46.042168261 +0000 UTC m=+0.675409586 container remove 12e88782f8473e9b4e0223fa1a608000cf363d0373b450fd99f0fd6c6b08189d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, release=754, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, distribution-scope=public, ceph=True, io.buildah.version=1.19.8) 2024-03-30T14:28:46.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi193.service: Deactivated successfully. 2024-03-30T14:28:46.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 systemd[1]: Stopped Ceph mon.smithi193 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:28:46.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi193.service: Consumed 7.148s CPU time. 2024-03-30T14:28:46.720 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 systemd[1]: Starting Ceph mon.smithi193 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:28:46.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 podman[52959]: 2024-03-30 14:28:46.595405406 +0000 UTC m=+0.021039244 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:28:46.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 podman[52959]: 2024-03-30 14:28:46.735477534 +0000 UTC m=+0.161111344 container create d6d2414c2ac116b4a095ff863a460a9ad4a40a1b4f79ad888157cd2e2dced53f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-03-30T14:28:47.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 154196 11.65 MB/sec execute 212 sec latency 62.216 ms 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 podman[52959]: 2024-03-30 14:28:46.988688656 +0000 UTC m=+0.414322487 container init d6d2414c2ac116b4a095ff863a460a9ad4a40a1b4f79ad888157cd2e2dced53f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:46 smithi193 podman[52959]: 2024-03-30 14:28:46.992506831 +0000 UTC m=+0.418140640 container start d6d2414c2ac116b4a095ff863a460a9ad4a40a1b4f79ad888157cd2e2dced53f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 2 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: pidfile_write: ignore empty --pid-file 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: load: jerasure load: lrc 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: RocksDB version: 7.9.2 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Git sha 0 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: DB SUMMARY 2024-03-30T14:28:47.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: DB Session ID: 5SATWSDV1WEY3QWLG2MV 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: CURRENT file: CURRENT 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi193/store.db dir, Total Num: 1, files: 000024.sst 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi193/store.db: 000022.log size: 7350248 ; 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.error_if_exists: 0 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.create_if_missing: 0 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.paranoid_checks: 1 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.env: 0x55a8e06b7c60 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T14:28:47.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.info_log: 0x55a8e29d79a0 2024-03-30T14:28:47.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.statistics: (nil) 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.use_fsync: 0 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_log_file_size: 0 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T14:28:47.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.allow_fallocate: 1 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.use_direct_reads: 0 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.db_log_dir: 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.wal_dir: 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T14:28:47.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.write_buffer_manager: 0x55a8e29aa1e0 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.unordered_write: 0 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T14:28:47.337 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.row_cache: None 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.wal_filter: None 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.two_write_queues: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.wal_compression: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.atomic_flush: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.log_readahead_size: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T14:28:47.338 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_background_jobs: 2 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_background_compactions: -1 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_subcompactions: 1 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T14:28:47.339 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_open_files: -1 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_background_flushes: -1 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Compression algorithms supported: 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kZSTD supported: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kXpressCompression supported: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kZlibCompression supported: 1 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kSnappyCompression supported: 1 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kLZ4Compression supported: 1 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: kBZip2Compression supported: 0 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T14:28:47.340 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi193/store.db/MANIFEST-000009 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.merge_operator: 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_filter: None 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a8e29d7aa0) 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: cache_index_and_filter_blocks: 1 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: pin_top_level_index_and_filter: 1 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: index_type: 0 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: data_block_index_type: 0 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: index_shortening: 1 2024-03-30T14:28:47.341 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: checksum: 4 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: no_block_cache: 0 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_cache: 0x55a8e29db090 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_cache_name: BinnedLRUCache 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_cache_options: 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: capacity : 536870912 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: num_shard_bits : 4 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: strict_capacity_limit : 0 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: high_pri_pool_ratio: 0.000 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_cache_compressed: (nil) 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: persistent_cache: (nil) 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_size: 4096 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_size_deviation: 10 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_restart_interval: 16 2024-03-30T14:28:47.342 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: index_block_restart_interval: 1 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: metadata_block_size: 4096 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: partition_filters: 0 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: use_delta_encoding: 1 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: filter_policy: bloomfilter 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: whole_key_filtering: 1 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: verify_compression: 0 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: read_amp_bytes_per_bit: 0 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: format_version: 5 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: enable_index_compression: 1 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: block_align: 0 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: max_auto_readahead_size: 262144 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: prepopulate_block_cache: 0 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: initial_auto_readahead_size: 8192 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T14:28:47.343 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression: NoCompression 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.num_levels: 7 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T14:28:47.344 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T14:28:47.345 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T14:28:47.346 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T14:28:47.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.table_properties_collectors: 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.inplace_update_support: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.bloom_locality: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.max_successive_merges: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T14:28:47.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.ttl: 2592000 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.enable_blob_files: false 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.min_blob_size: 0 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-03-30T14:28:47.349 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi193/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12266, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dd57e9cd-2f8f-4844-953c-eeeb1cf229ce 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808927025172, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808927043335, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4493197, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12260, "largest_seqno": 14079, "table_properties": {"data_size": 4485262, "index_size": 5059, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1861, "raw_key_size": 16717, "raw_average_key_size": 23, "raw_value_size": 4469715, "raw_average_value_size": 6190, "num_data_blocks": 229, "num_entries": 722, "num_filter_entries": 722, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711808927, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dd57e9cd-2f8f-4844-953c-eeeb1cf229ce", "db_session_id": "5SATWSDV1WEY3QWLG2MV", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711808927043475, "job": 1, "event": "recovery_finished"} 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi193/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a8e2acc000 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: DB pointer 0x55a8e2ab6000 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T14:28:47.350 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: ** DB Stats ** 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: ** Compaction Stats [default] ** 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: L0 1/0 4.29 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-30T14:28:47.351 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: L6 1/0 7.02 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Sum 2/0 11.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: ** Compaction Stats [default] ** 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 530.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T14:28:47.352 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Cumulative compaction: 0.00 GB write, 185.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Interval compaction: 0.00 GB write, 185.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Block cache BinnedLRUCache@0x55a8e29db090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: starting mon.smithi193 rank 1 at public addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] at bind addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi193 fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???) e2 preinit fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???).mds e7 new map 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???).mds e7 print_map 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: e7 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T14:28:47.353 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: legacy client fscid: 1 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Filesystem 'foofs' (1) 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: fs_name foofs 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: epoch 5 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: created 2024-03-30T14:22:35.894607+0000 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: modified 2024-03-30T14:22:41.925357+0000 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: tableserver 0 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: root 0 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: session_timeout 60 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: session_autoclose 300 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: max_file_size 1099511627776 2024-03-30T14:28:47.354 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: max_xattr_size 65536 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: required_client_features {} 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: last_failure 0 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: last_failure_osd_epoch 0 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: max_mds 1 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: in 0 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: up {0=14498} 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: failed 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: damaged 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: stopped 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: data_pools [3] 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: metadata_pool 2 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: inline_data disabled 2024-03-30T14:28:47.355 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: balancer 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: bal_rank_mask -1 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: standby_count_wanted 1 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: qdb_cluster leader: 0 members: 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: [mds.foofs.smithi031.xsynqy{0:14498} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.31:6834/669631789,v1:172.21.15.31:6835/669631789] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: Standby daemons: 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout: [mds.foofs.smithi193.shgxxs{-1:24301} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-30T14:28:47.356 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 ceph-mon[52977]: mon.smithi193@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-03-30T14:28:47.357 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 bash[52959]: d6d2414c2ac116b4a095ff863a460a9ad4a40a1b4f79ad888157cd2e2dced53f 2024-03-30T14:28:47.357 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:47 smithi193 systemd[1]: Started Ceph mon.smithi193 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:28:48.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 154620 11.63 MB/sec execute 213 sec latency 46.597 ms 2024-03-30T14:28:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 154966 11.62 MB/sec execute 214 sec latency 53.222 ms 2024-03-30T14:28:49.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:28:49.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: mgrmap e42: smithi031.lkrchi(active, since 21s), standbys: smithi193.ikzlnb 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: mon.smithi031 calling monitor election 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: mon.smithi193 calling monitor election 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: mon.smithi031 is new leader, mons smithi031,smithi193 in quorum (ranks 0,1) 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: monmap epoch 3 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:49.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: last_changed 2024-03-30T14:28:48.500571+0000 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: created 2024-03-30T14:15:16.778426+0000 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: min_mon_release 19 (squid) 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: election_strategy: 1 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.smithi031 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.smithi193 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: mgrmap e42: smithi031.lkrchi(active, since 21s), standbys: smithi193.ikzlnb 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:49.690 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:49.691 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:28:49.691 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:49.691 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:49.691 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: mgrmap e42: smithi031.lkrchi(active, since 21s), standbys: smithi193.ikzlnb 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: mon.smithi031 calling monitor election 2024-03-30T14:28:49.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: mon.smithi193 calling monitor election 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: mon.smithi031 is new leader, mons smithi031,smithi193 in quorum (ranks 0,1) 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: monmap epoch 3 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: last_changed 2024-03-30T14:28:48.500571+0000 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: created 2024-03-30T14:15:16.778426+0000 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: min_mon_release 19 (squid) 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: election_strategy: 1 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.smithi031 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.smithi193 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:28:49.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: mgrmap e42: smithi031.lkrchi(active, since 21s), standbys: smithi193.ikzlnb 2024-03-30T14:28:49.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:28:49.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:28:49.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:28:49.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:49.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:49.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 155512 11.64 MB/sec execute 215 sec latency 31.585 ms 2024-03-30T14:28:50.693 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:50 smithi031 ceph-mon[67456]: pgmap v14: 97 pgs: 97 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 800 op/s 2024-03-30T14:28:50.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:50 smithi193 ceph-mon[52977]: pgmap v14: 97 pgs: 97 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 800 op/s 2024-03-30T14:28:51.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 156009 11.67 MB/sec execute 216 sec latency 41.745 ms 2024-03-30T14:28:52.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 156491 11.69 MB/sec execute 217 sec latency 22.767 ms 2024-03-30T14:28:52.696 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:52 smithi031 ceph-mon[67456]: pgmap v15: 97 pgs: 97 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.2 MiB/s wr, 524 op/s 2024-03-30T14:28:52.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:52 smithi193 ceph-mon[52977]: pgmap v15: 97 pgs: 97 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.2 MiB/s wr, 524 op/s 2024-03-30T14:28:53.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 157012 11.68 MB/sec execute 218 sec latency 40.598 ms 2024-03-30T14:28:53.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:53 smithi031 ceph-mon[67456]: pgmap v16: 97 pgs: 97 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 485 op/s 2024-03-30T14:28:53.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:53.706 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:54.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:53 smithi193 ceph-mon[52977]: pgmap v16: 97 pgs: 97 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 485 op/s 2024-03-30T14:28:54.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:54.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:54.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 157366 11.67 MB/sec execute 219 sec latency 32.230 ms 2024-03-30T14:28:54.964 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:54.964 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:54.967 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:54.968 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:55.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 157847 11.66 MB/sec execute 220 sec latency 43.046 ms 2024-03-30T14:28:56.000 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:55 smithi031 ceph-mon[67456]: pgmap v17: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.0 MiB/s wr, 777 op/s 2024-03-30T14:28:56.000 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:55 smithi193 ceph-mon[52977]: pgmap v17: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.0 MiB/s wr, 777 op/s 2024-03-30T14:28:56.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 158283 11.66 MB/sec execute 221 sec latency 59.542 ms 2024-03-30T14:28:57.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 158671 11.66 MB/sec execute 222 sec latency 33.927 ms 2024-03-30T14:28:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 158998 11.65 MB/sec execute 223 sec latency 51.250 ms 2024-03-30T14:28:58.784 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: pgmap v18: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.4 MiB/s wr, 544 op/s 2024-03-30T14:28:58.784 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.784 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.784 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:58.784 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:58.785 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.785 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.785 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:58.785 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:28:58.785 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:28:58.785 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: pgmap v18: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.4 MiB/s wr, 544 op/s 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:28:58.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:28:58.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:59.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 159407 11.65 MB/sec execute 224 sec latency 36.645 ms 2024-03-30T14:28:59.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:59 smithi031 ceph-mon[67456]: pgmap v19: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 593 op/s 2024-03-30T14:28:59.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:59 smithi031 ceph-mon[67456]: pgmap v20: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 389 op/s 2024-03-30T14:28:59.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:59 smithi031 ceph-mon[67456]: Reconfiguring mon.smithi031 (monmap changed)... 2024-03-30T14:28:59.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:28:59 smithi031 ceph-mon[67456]: Reconfiguring daemon mon.smithi031 on smithi031 2024-03-30T14:28:59.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:59 smithi193 ceph-mon[52977]: pgmap v19: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 593 op/s 2024-03-30T14:28:59.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:59 smithi193 ceph-mon[52977]: pgmap v20: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 389 op/s 2024-03-30T14:28:59.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:59 smithi193 ceph-mon[52977]: Reconfiguring mon.smithi031 (monmap changed)... 2024-03-30T14:28:59.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:28:59 smithi193 ceph-mon[52977]: Reconfiguring daemon mon.smithi031 on smithi031 2024-03-30T14:29:00.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 159669 11.64 MB/sec execute 225 sec latency 39.269 ms 2024-03-30T14:29:00.820 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:00 smithi031 ceph-mon[67456]: pgmap v21: 97 pgs: 97 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 720 op/s 2024-03-30T14:29:00.820 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:00.821 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:00.821 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:29:00.821 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:29:00.821 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:00.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:00 smithi193 ceph-mon[52977]: pgmap v21: 97 pgs: 97 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 720 op/s 2024-03-30T14:29:00.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:00.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:00.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.lkrchi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:29:00.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:29:00.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 159956 11.63 MB/sec execute 226 sec latency 48.836 ms 2024-03-30T14:29:01.840 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:01 smithi031 ceph-mon[67456]: Reconfiguring mgr.smithi031.lkrchi (monmap changed)... 2024-03-30T14:29:01.840 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:01 smithi031 ceph-mon[67456]: Reconfiguring daemon mgr.smithi031.lkrchi on smithi031 2024-03-30T14:29:02.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:01 smithi193 ceph-mon[52977]: Reconfiguring mgr.smithi031.lkrchi (monmap changed)... 2024-03-30T14:29:02.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:01 smithi193 ceph-mon[52977]: Reconfiguring daemon mgr.smithi031.lkrchi on smithi031 2024-03-30T14:29:02.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 160259 11.61 MB/sec execute 227 sec latency 61.850 ms 2024-03-30T14:29:02.600 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:02 smithi031 ceph-mon[67456]: pgmap v22: 97 pgs: 97 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 720 op/s 2024-03-30T14:29:03.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:02 smithi193 ceph-mon[52977]: pgmap v22: 97 pgs: 97 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.3 MiB/s wr, 720 op/s 2024-03-30T14:29:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 160683 11.60 MB/sec execute 228 sec latency 38.300 ms 2024-03-30T14:29:03.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:03 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:03.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:03 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:03.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:03 smithi031 ceph-mon[67456]: Reconfiguring crash.smithi031 (monmap changed)... 2024-03-30T14:29:03.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:03 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:03.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:03 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:03.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:03 smithi031 ceph-mon[67456]: Reconfiguring daemon crash.smithi031 on smithi031 2024-03-30T14:29:03.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:03 smithi031 ceph-mon[67456]: pgmap v23: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.1 MiB/s wr, 608 op/s 2024-03-30T14:29:04.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:03 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:04.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:03 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:04.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:03 smithi193 ceph-mon[52977]: Reconfiguring crash.smithi031 (monmap changed)... 2024-03-30T14:29:04.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:03 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:04.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:03 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:04.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:03 smithi193 ceph-mon[52977]: Reconfiguring daemon crash.smithi031 on smithi031 2024-03-30T14:29:04.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:03 smithi193 ceph-mon[52977]: pgmap v23: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.1 MiB/s wr, 608 op/s 2024-03-30T14:29:04.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 161093 11.61 MB/sec execute 229 sec latency 37.016 ms 2024-03-30T14:29:05.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 161689 11.60 MB/sec execute 230 sec latency 45.532 ms 2024-03-30T14:29:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 162322 11.64 MB/sec execute 231 sec latency 36.353 ms 2024-03-30T14:29:06.375 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:06.375 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:06.375 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:06 smithi031 ceph-mon[67456]: Reconfiguring osd.0 (monmap changed)... 2024-03-30T14:29:06.375 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:29:06.375 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:06.375 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:06 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.0 on smithi031 2024-03-30T14:29:06.375 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:06 smithi031 ceph-mon[67456]: pgmap v24: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.1 MiB/s wr, 608 op/s 2024-03-30T14:29:06.581 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:06.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:06.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:06 smithi193 ceph-mon[52977]: Reconfiguring osd.0 (monmap changed)... 2024-03-30T14:29:06.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:29:06.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:06.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:06 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.0 on smithi031 2024-03-30T14:29:06.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:06 smithi193 ceph-mon[52977]: pgmap v24: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.1 MiB/s wr, 608 op/s 2024-03-30T14:29:07.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 162847 11.64 MB/sec execute 232 sec latency 41.610 ms 2024-03-30T14:29:07.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:07 smithi031 ceph-mon[67456]: pgmap v25: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-03-30T14:29:08.002 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:07 smithi193 ceph-mon[52977]: pgmap v25: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-03-30T14:29:08.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 163266 11.66 MB/sec execute 233 sec latency 62.081 ms 2024-03-30T14:29:09.135 INFO:teuthology.orchestra.run.smithi031.stdout: 5 163719 11.66 MB/sec execute 234 sec latency 30.653 ms 2024-03-30T14:29:09.651 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:09 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:09.651 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:09 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:09.651 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:09 smithi031 ceph-mon[67456]: Reconfiguring osd.1 (monmap changed)... 2024-03-30T14:29:09.651 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:09 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:29:09.652 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:09 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:09.652 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:09 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.1 on smithi031 2024-03-30T14:29:09.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:09 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:09.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:09 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:09.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:09 smithi193 ceph-mon[52977]: Reconfiguring osd.1 (monmap changed)... 2024-03-30T14:29:09.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:09 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:29:09.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:09 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:09.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:09 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.1 on smithi031 2024-03-30T14:29:10.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 164115 11.65 MB/sec execute 235 sec latency 70.228 ms 2024-03-30T14:29:10.659 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:10 smithi031 ceph-mon[67456]: pgmap v26: 97 pgs: 97 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 775 op/s 2024-03-30T14:29:10.831 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:10 smithi193 ceph-mon[52977]: pgmap v26: 97 pgs: 97 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 775 op/s 2024-03-30T14:29:11.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 164380 11.63 MB/sec execute 236 sec latency 46.268 ms 2024-03-30T14:29:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 164673 11.62 MB/sec execute 237 sec latency 58.864 ms 2024-03-30T14:29:12.182 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:12.182 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:12.182 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:12 smithi031 ceph-mon[67456]: Reconfiguring osd.2 (monmap changed)... 2024-03-30T14:29:12.182 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:29:12.182 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:12.183 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:12 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.2 on smithi031 2024-03-30T14:29:12.183 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:12 smithi031 ceph-mon[67456]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-03-30T14:29:12.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:12.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:12.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:12 smithi193 ceph-mon[52977]: Reconfiguring osd.2 (monmap changed)... 2024-03-30T14:29:12.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:29:12.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:12.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:12 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.2 on smithi031 2024-03-30T14:29:12.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:12 smithi193 ceph-mon[52977]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-03-30T14:29:13.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 165003 11.59 MB/sec execute 238 sec latency 48.420 ms 2024-03-30T14:29:13.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:13.326 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:29:13.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:14.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 165266 11.57 MB/sec execute 239 sec latency 41.497 ms 2024-03-30T14:29:14.216 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:14 smithi031 ceph-mon[67456]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:14.216 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:14 smithi031 ceph-mon[67456]: pgmap v28: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 738 op/s 2024-03-30T14:29:14.216 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:14 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:14.216 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:14 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:14.217 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:14 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:29:14.217 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:14 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:14.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:14 smithi193 ceph-mon[52977]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:14.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:14 smithi193 ceph-mon[52977]: pgmap v28: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 738 op/s 2024-03-30T14:29:14.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:14 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:14.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:14 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:14.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:14 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:29:14.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:14 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:14.458 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (3m) 37s ago 12m 16.6M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (12m) 37s ago 12m 6395k - 17.2.0 e1d6a67b021e 5c67003ce807 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (11m) 21s ago 11m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 37s ago 12m - - 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (6m) 37s ago 6m 40.0M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (6m) 21s ago 6m 14.6M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (2m) 37s ago 13m 477M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (81s) 21s ago 11m 448M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (61s) 37s ago 13m 31.7M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (27s) 21s ago 11m 25.0M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (6m) 37s ago 6m 272M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (6m) 21s ago 6m 82.4M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:29:14.459 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (3m) 37s ago 12m 12.0M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (2m) 21s ago 11m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (11m) 37s ago 11m 523M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (10m) 37s ago 10m 453M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (10m) 37s ago 10m 389M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (9m) 37s ago 9m 487M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (9m) 21s ago 9m 513M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (8m) 21s ago 8m 326M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (8m) 21s ago 8m 519M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (8m) 21s ago 8m 359M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:29:14.460 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (88s) 37s ago 11m 55.4M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:29:15.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:15.073 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:29:15.073 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-30T14:29:15.073 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 4 2024-03-30T14:29:15.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:29:15.073 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:29:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 165504 11.56 MB/sec execute 240 sec latency 68.138 ms 2024-03-30T14:29:15.252 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:15 smithi031 ceph-mon[67456]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:15.252 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:15 smithi031 ceph-mon[67456]: Reconfiguring osd.3 (monmap changed)... 2024-03-30T14:29:15.252 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:15 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.3 on smithi031 2024-03-30T14:29:15.252 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:15 smithi031 ceph-mon[67456]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:15.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:15 smithi193 ceph-mon[52977]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:15.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:15 smithi193 ceph-mon[52977]: Reconfiguring osd.3 (monmap changed)... 2024-03-30T14:29:15.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:15 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.3 on smithi031 2024-03-30T14:29:15.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:15 smithi193 ceph-mon[52977]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:15.577 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:29:15.577 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:29:15.577 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:29:15.578 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:29:15.578 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:29:15.578 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:29:15.578 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:29:15.578 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:29:15.578 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "4/23 daemons upgraded", 2024-03-30T14:29:15.578 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading mon daemons", 2024-03-30T14:29:15.579 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:29:15.579 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:29:16.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 165773 11.54 MB/sec execute 241 sec latency 74.230 ms 2024-03-30T14:29:16.194 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:29:16.194 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:29:16.194 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:29:16.280 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:16 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/3451524629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:16.281 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:16 smithi031 ceph-mon[67456]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:16.281 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:16 smithi031 ceph-mon[67456]: pgmap v29: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 530 op/s 2024-03-30T14:29:16.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:16 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/3451524629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:16.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:16 smithi193 ceph-mon[52977]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:16.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:16 smithi193 ceph-mon[52977]: pgmap v29: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 530 op/s 2024-03-30T14:29:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 166049 11.53 MB/sec execute 242 sec latency 64.000 ms 2024-03-30T14:29:17.314 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:17 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2971256656' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:29:17.314 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:17 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:17.314 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:17 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:17.315 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:17 smithi031 ceph-mon[67456]: Reconfiguring mds.foofs.smithi031.xsynqy (monmap changed)... 2024-03-30T14:29:17.315 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:17 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:29:17.315 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:17 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:17.315 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:17 smithi031 ceph-mon[67456]: Reconfiguring daemon mds.foofs.smithi031.xsynqy on smithi031 2024-03-30T14:29:17.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:17 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2971256656' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:29:17.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:17 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:17.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:17 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:17.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:17 smithi193 ceph-mon[52977]: Reconfiguring mds.foofs.smithi031.xsynqy (monmap changed)... 2024-03-30T14:29:17.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:17 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:29:17.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:17 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:17.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:17 smithi193 ceph-mon[52977]: Reconfiguring daemon mds.foofs.smithi031.xsynqy on smithi031 2024-03-30T14:29:18.089 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:18 smithi031 ceph-mon[67456]: pgmap v30: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 530 op/s 2024-03-30T14:29:18.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 166371 11.51 MB/sec execute 243 sec latency 52.009 ms 2024-03-30T14:29:18.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:18 smithi193 ceph-mon[52977]: pgmap v30: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 530 op/s 2024-03-30T14:29:19.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 166677 11.51 MB/sec execute 244 sec latency 57.589 ms 2024-03-30T14:29:20.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 166895 11.48 MB/sec execute 245 sec latency 47.998 ms 2024-03-30T14:29:20.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:20.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:20.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:20 smithi031 ceph-mon[67456]: Reconfiguring crash.smithi193 (monmap changed)... 2024-03-30T14:29:20.631 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:20.632 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:20.632 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:20 smithi031 ceph-mon[67456]: Reconfiguring daemon crash.smithi193 on smithi193 2024-03-30T14:29:20.632 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:20 smithi031 ceph-mon[67456]: pgmap v31: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 701 op/s 2024-03-30T14:29:20.775 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:20.775 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:20.775 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:20 smithi193 ceph-mon[52977]: Reconfiguring crash.smithi193 (monmap changed)... 2024-03-30T14:29:20.775 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:20.776 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:20.776 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:20 smithi193 ceph-mon[52977]: Reconfiguring daemon crash.smithi193 on smithi193 2024-03-30T14:29:20.776 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:20 smithi193 ceph-mon[52977]: pgmap v31: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 701 op/s 2024-03-30T14:29:21.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 167204 11.48 MB/sec execute 246 sec latency 62.974 ms 2024-03-30T14:29:21.647 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:21 smithi031 ceph-mon[67456]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-03-30T14:29:21.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:21 smithi193 ceph-mon[52977]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-03-30T14:29:22.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 167536 11.45 MB/sec execute 247 sec latency 44.965 ms 2024-03-30T14:29:22.915 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:22.915 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:22.916 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:22 smithi031 ceph-mon[67456]: Reconfiguring mgr.smithi193.ikzlnb (monmap changed)... 2024-03-30T14:29:22.916 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:29:22.916 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:29:22.916 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:22.916 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:22 smithi031 ceph-mon[67456]: Reconfiguring daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:29:23.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:23.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:23.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:22 smithi193 ceph-mon[52977]: Reconfiguring mgr.smithi193.ikzlnb (monmap changed)... 2024-03-30T14:29:23.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.ikzlnb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:29:23.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:29:23.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:23.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:22 smithi193 ceph-mon[52977]: Reconfiguring daemon mgr.smithi193.ikzlnb on smithi193 2024-03-30T14:29:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 167789 11.44 MB/sec execute 248 sec latency 41.611 ms 2024-03-30T14:29:24.069 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:23 smithi031 ceph-mon[67456]: pgmap v33: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 558 op/s 2024-03-30T14:29:24.070 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:24.070 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:24.070 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:29:24.070 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:29:24.070 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:24.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:23 smithi193 ceph-mon[52977]: pgmap v33: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 558 op/s 2024-03-30T14:29:24.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:24.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:24.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:29:24.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:29:24.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 168057 11.42 MB/sec execute 249 sec latency 63.270 ms 2024-03-30T14:29:25.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:24 smithi031 ceph-mon[67456]: Reconfiguring mon.smithi193 (monmap changed)... 2024-03-30T14:29:25.080 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:24 smithi031 ceph-mon[67456]: Reconfiguring daemon mon.smithi193 on smithi193 2024-03-30T14:29:25.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:24 smithi193 ceph-mon[52977]: Reconfiguring mon.smithi193 (monmap changed)... 2024-03-30T14:29:25.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:24 smithi193 ceph-mon[52977]: Reconfiguring daemon mon.smithi193 on smithi193 2024-03-30T14:29:25.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 168362 11.40 MB/sec execute 250 sec latency 66.009 ms 2024-03-30T14:29:26.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 168632 11.38 MB/sec execute 251 sec latency 58.252 ms 2024-03-30T14:29:26.649 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:26.649 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:26.649 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:26 smithi031 ceph-mon[67456]: Reconfiguring osd.4 (monmap changed)... 2024-03-30T14:29:26.650 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:29:26.650 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:26.650 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:26 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.4 on smithi193 2024-03-30T14:29:26.650 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:26 smithi031 ceph-mon[67456]: pgmap v34: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.6 MiB/s wr, 347 op/s 2024-03-30T14:29:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:26 smithi193 ceph-mon[52977]: Reconfiguring osd.4 (monmap changed)... 2024-03-30T14:29:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:29:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:26 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.4 on smithi193 2024-03-30T14:29:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:26 smithi193 ceph-mon[52977]: pgmap v34: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.6 MiB/s wr, 347 op/s 2024-03-30T14:29:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 168919 11.37 MB/sec execute 252 sec latency 56.919 ms 2024-03-30T14:29:28.136 INFO:teuthology.orchestra.run.smithi031.stdout: 5 169112 11.35 MB/sec execute 253 sec latency 70.301 ms 2024-03-30T14:29:28.434 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:28.434 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:28.434 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: Reconfiguring osd.5 (monmap changed)... 2024-03-30T14:29:28.434 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:29:28.434 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:28.435 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.5 on smithi193 2024-03-30T14:29:28.435 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: pgmap v35: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.6 MiB/s wr, 347 op/s 2024-03-30T14:29:28.435 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:28.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:28.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:28.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: Reconfiguring osd.5 (monmap changed)... 2024-03-30T14:29:28.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:29:28.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:28.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.5 on smithi193 2024-03-30T14:29:28.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: pgmap v35: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.6 MiB/s wr, 347 op/s 2024-03-30T14:29:28.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:29.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 169528 11.35 MB/sec execute 254 sec latency 45.157 ms 2024-03-30T14:29:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 169939 11.35 MB/sec execute 255 sec latency 26.029 ms 2024-03-30T14:29:30.198 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:30 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:30.198 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:30 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:30.198 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:30 smithi031 ceph-mon[67456]: Reconfiguring osd.6 (monmap changed)... 2024-03-30T14:29:30.199 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:30 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:29:30.199 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:30 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:30.199 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:30 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.6 on smithi193 2024-03-30T14:29:30.199 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:30 smithi031 ceph-mon[67456]: pgmap v36: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.1 MiB/s wr, 518 op/s 2024-03-30T14:29:30.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:30 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:30.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:30 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:30.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:30 smithi193 ceph-mon[52977]: Reconfiguring osd.6 (monmap changed)... 2024-03-30T14:29:30.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:30 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:29:30.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:30 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:30.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:30 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.6 on smithi193 2024-03-30T14:29:30.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:30 smithi193 ceph-mon[52977]: pgmap v36: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.1 MiB/s wr, 518 op/s 2024-03-30T14:29:31.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 170309 11.35 MB/sec execute 256 sec latency 42.387 ms 2024-03-30T14:29:31.964 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:31.964 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:31.965 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:31 smithi031 ceph-mon[67456]: Reconfiguring osd.7 (monmap changed)... 2024-03-30T14:29:31.965 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:29:31.965 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:31.965 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:31 smithi031 ceph-mon[67456]: Reconfiguring daemon osd.7 on smithi193 2024-03-30T14:29:31.965 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:31 smithi031 ceph-mon[67456]: pgmap v37: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 347 op/s 2024-03-30T14:29:32.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 170691 11.36 MB/sec execute 257 sec latency 35.469 ms 2024-03-30T14:29:32.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:32.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:32.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:31 smithi193 ceph-mon[52977]: Reconfiguring osd.7 (monmap changed)... 2024-03-30T14:29:32.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:29:32.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:32.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:31 smithi193 ceph-mon[52977]: Reconfiguring daemon osd.7 on smithi193 2024-03-30T14:29:32.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:31 smithi193 ceph-mon[52977]: pgmap v37: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 347 op/s 2024-03-30T14:29:33.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 171011 11.33 MB/sec execute 258 sec latency 48.705 ms 2024-03-30T14:29:33.744 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:33.745 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:33.745 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:33 smithi031 ceph-mon[67456]: Reconfiguring mds.foofs.smithi193.shgxxs (monmap changed)... 2024-03-30T14:29:33.745 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:29:33.745 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:33.745 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:33 smithi031 ceph-mon[67456]: Reconfiguring daemon mds.foofs.smithi193.shgxxs on smithi193 2024-03-30T14:29:33.745 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:33 smithi031 ceph-mon[67456]: pgmap v38: 97 pgs: 97 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 568 op/s 2024-03-30T14:29:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:33 smithi193 ceph-mon[52977]: Reconfiguring mds.foofs.smithi193.shgxxs (monmap changed)... 2024-03-30T14:29:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:29:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:33 smithi193 ceph-mon[52977]: Reconfiguring daemon mds.foofs.smithi193.shgxxs on smithi193 2024-03-30T14:29:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:33 smithi193 ceph-mon[52977]: pgmap v38: 97 pgs: 97 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 568 op/s 2024-03-30T14:29:34.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 171292 11.32 MB/sec execute 259 sec latency 45.953 ms 2024-03-30T14:29:35.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 171511 11.30 MB/sec execute 260 sec latency 71.834 ms 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all mon 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi031"}]: dispatch 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi031"}]': finished 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi193"}]: dispatch 2024-03-30T14:29:35.528 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi193"}]': finished 2024-03-30T14:29:35.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:35.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:35.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:29:35.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:35.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:35.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all mon 2024-03-30T14:29:35.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:35.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi031"}]: dispatch 2024-03-30T14:29:35.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi031"}]': finished 2024-03-30T14:29:35.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi193"}]: dispatch 2024-03-30T14:29:35.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi193"}]': finished 2024-03-30T14:29:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 171836 11.29 MB/sec execute 261 sec latency 73.309 ms 2024-03-30T14:29:36.540 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:36 smithi031 ceph-mon[67456]: pgmap v39: 97 pgs: 97 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 392 op/s 2024-03-30T14:29:36.733 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:36 smithi193 ceph-mon[52977]: pgmap v39: 97 pgs: 97 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 392 op/s 2024-03-30T14:29:37.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 172174 11.27 MB/sec execute 262 sec latency 50.290 ms 2024-03-30T14:29:38.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:37 smithi031 ceph-mon[67456]: Upgrade: Updating crash.smithi031 (1/2) 2024-03-30T14:29:38.085 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:37 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:38.085 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:37 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:38.085 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:37 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:38.085 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:37 smithi031 ceph-mon[67456]: Deploying daemon crash.smithi031 on smithi031 2024-03-30T14:29:38.085 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:37 smithi031 ceph-mon[67456]: pgmap v40: 97 pgs: 97 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 392 op/s 2024-03-30T14:29:38.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 172458 11.26 MB/sec execute 263 sec latency 50.284 ms 2024-03-30T14:29:38.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:37 smithi193 ceph-mon[52977]: Upgrade: Updating crash.smithi031 (1/2) 2024-03-30T14:29:38.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:37 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:38.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:37 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:38.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:37 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:38.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:37 smithi193 ceph-mon[52977]: Deploying daemon crash.smithi031 on smithi031 2024-03-30T14:29:38.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:37 smithi193 ceph-mon[52977]: pgmap v40: 97 pgs: 97 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 392 op/s 2024-03-30T14:29:39.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 172752 11.25 MB/sec execute 264 sec latency 73.066 ms 2024-03-30T14:29:39.730 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:39 smithi031 ceph-mon[67456]: pgmap v41: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.6 MiB/s wr, 577 op/s 2024-03-30T14:29:40.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:39 smithi193 ceph-mon[52977]: pgmap v41: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.6 MiB/s wr, 577 op/s 2024-03-30T14:29:40.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 173314 11.27 MB/sec execute 265 sec latency 28.959 ms 2024-03-30T14:29:41.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 173817 11.28 MB/sec execute 266 sec latency 38.076 ms 2024-03-30T14:29:41.671 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:41 smithi031 ceph-mon[67456]: pgmap v42: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 406 op/s 2024-03-30T14:29:42.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:41 smithi193 ceph-mon[52977]: pgmap v42: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 406 op/s 2024-03-30T14:29:42.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 174294 11.29 MB/sec execute 267 sec latency 37.127 ms 2024-03-30T14:29:42.685 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:43.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:43.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 174729 11.28 MB/sec execute 268 sec latency 29.766 ms 2024-03-30T14:29:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 175086 11.28 MB/sec execute 269 sec latency 41.168 ms 2024-03-30T14:29:44.698 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:44 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:44.698 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:44 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:44.698 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:44 smithi031 ceph-mon[67456]: pgmap v43: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.9 MiB/s wr, 680 op/s 2024-03-30T14:29:44.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:44 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:44.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:44 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:44.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:44 smithi193 ceph-mon[52977]: pgmap v43: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.9 MiB/s wr, 680 op/s 2024-03-30T14:29:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 175607 11.28 MB/sec execute 270 sec latency 33.956 ms 2024-03-30T14:29:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 176075 11.29 MB/sec execute 271 sec latency 59.255 ms 2024-03-30T14:29:46.484 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:46 smithi031 ceph-mon[67456]: Upgrade: Updating crash.smithi193 (2/2) 2024-03-30T14:29:46.484 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:46 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:46.484 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:46 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:46.484 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:46 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:46.484 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:46 smithi031 ceph-mon[67456]: Deploying daemon crash.smithi193 on smithi193 2024-03-30T14:29:46.484 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:46 smithi031 ceph-mon[67456]: pgmap v44: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-03-30T14:29:46.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:46 smithi193 ceph-mon[52977]: Upgrade: Updating crash.smithi193 (2/2) 2024-03-30T14:29:46.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:46 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:46.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:46 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T14:29:46.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:46 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:46.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:46 smithi193 ceph-mon[52977]: Deploying daemon crash.smithi193 on smithi193 2024-03-30T14:29:46.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:46 smithi193 ceph-mon[52977]: pgmap v44: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-03-30T14:29:46.847 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:29:47.137 INFO:teuthology.orchestra.run.smithi031.stdout: 5 176523 11.29 MB/sec execute 272 sec latency 53.215 ms 2024-03-30T14:29:47.738 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:47 smithi031 ceph-mon[67456]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:47.739 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:47 smithi031 ceph-mon[67456]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:47.739 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:47 smithi031 ceph-mon[67456]: pgmap v45: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-03-30T14:29:47.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:47 smithi193 ceph-mon[52977]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:47.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:47 smithi193 ceph-mon[52977]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:47.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:47 smithi193 ceph-mon[52977]: pgmap v45: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-03-30T14:29:47.970 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:29:47.970 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (3m) 70s ago 13m 16.6M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 starting - - - - 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (12m) 55s ago 12m 6405k - 17.2.0 e1d6a67b021e 7342e032481b 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 70s ago 12m - - 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (7m) 70s ago 7m 40.0M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (7m) 55s ago 7m 14.6M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (2m) 70s ago 14m 477M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (114s) 55s ago 12m 448M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (94s) 70s ago 14m 31.7M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (61s) 55s ago 12m 25.0M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (6m) 70s ago 6m 272M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (6m) 55s ago 6m 82.4M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (3m) 70s ago 12m 12.0M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (3m) 55s ago 12m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (11m) 70s ago 11m 523M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:29:47.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (11m) 70s ago 11m 453M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:29:47.972 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (10m) 70s ago 10m 389M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:29:47.972 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (10m) 70s ago 10m 487M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:29:47.972 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (9m) 55s ago 9m 513M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:29:47.972 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (9m) 55s ago 9m 326M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:29:47.972 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (9m) 55s ago 9m 519M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:29:47.972 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (8m) 55s ago 8m 359M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:29:47.972 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (2m) 70s ago 11m 55.4M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:29:48.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 176955 11.29 MB/sec execute 273 sec latency 55.110 ms 2024-03-30T14:29:48.694 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:29:48.694 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:29:48.694 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:29:48.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:48.694 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:29:48.694 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:29:48.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 4 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:29:48.695 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:29:48.748 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:48 smithi031 ceph-mon[67456]: from='client.44133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:48.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:48 smithi193 ceph-mon[52977]: from='client.44133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 177322 11.30 MB/sec execute 274 sec latency 37.473 ms 2024-03-30T14:29:49.302 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:29:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:29:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:29:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "4/23 daemons upgraded", 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading crash daemons", 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:29:49.303 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:29:49.758 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:49 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/999837593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:49.759 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:49.759 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:49.759 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:29:49.759 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:49 smithi031 ceph-mon[67456]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:49.759 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:49 smithi031 ceph-mon[67456]: pgmap v46: 97 pgs: 97 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 737 op/s 2024-03-30T14:29:49.956 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:29:49.956 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:29:49.956 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:29:50.068 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:49 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/999837593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:29:50.069 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:50.069 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:50.069 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:29:50.069 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:49 smithi193 ceph-mon[52977]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:29:50.069 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:49 smithi193 ceph-mon[52977]: pgmap v46: 97 pgs: 97 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 737 op/s 2024-03-30T14:29:50.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 177609 11.28 MB/sec execute 275 sec latency 102.605 ms 2024-03-30T14:29:50.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:50 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/848445863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:29:50.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:50 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/848445863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:29:51.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 178016 11.28 MB/sec execute 276 sec latency 46.743 ms 2024-03-30T14:29:51.794 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:51 smithi031 ceph-mon[67456]: pgmap v47: 97 pgs: 97 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 551 op/s 2024-03-30T14:29:51.930 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:51 smithi193 ceph-mon[52977]: pgmap v47: 97 pgs: 97 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 551 op/s 2024-03-30T14:29:52.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 178367 11.27 MB/sec execute 277 sec latency 52.979 ms 2024-03-30T14:29:53.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 178676 11.26 MB/sec execute 278 sec latency 48.267 ms 2024-03-30T14:29:54.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 179025 11.25 MB/sec execute 279 sec latency 47.306 ms 2024-03-30T14:29:54.538 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:54.538 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:54.538 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:54 smithi031 ceph-mon[67456]: pgmap v48: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 766 op/s 2024-03-30T14:29:54.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:54.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:54.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:54 smithi193 ceph-mon[52977]: pgmap v48: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 766 op/s 2024-03-30T14:29:55.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 179342 11.24 MB/sec execute 280 sec latency 55.699 ms 2024-03-30T14:29:55.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:55.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:55.811 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:55 smithi031 ceph-mon[67456]: pgmap v49: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-03-30T14:29:56.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:56.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:56.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:55 smithi193 ceph-mon[52977]: pgmap v49: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-03-30T14:29:56.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 179804 11.25 MB/sec execute 281 sec latency 38.845 ms 2024-03-30T14:29:57.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 180369 11.26 MB/sec execute 282 sec latency 56.527 ms 2024-03-30T14:29:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 180684 11.25 MB/sec execute 283 sec latency 53.888 ms 2024-03-30T14:29:58.619 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:58.619 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:58.619 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:58.619 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:58 smithi031 ceph-mon[67456]: pgmap v50: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-03-30T14:29:58.819 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:58.820 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:58.820 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:58.820 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:58 smithi193 ceph-mon[52977]: pgmap v50: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-03-30T14:29:59.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 180927 11.24 MB/sec execute 284 sec latency 45.900 ms 2024-03-30T14:29:59.924 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:59 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:59.924 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:59 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:59.924 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:59 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:59.924 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:59 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:29:59.924 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:29:59 smithi031 ceph-mon[67456]: pgmap v51: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 732 op/s 2024-03-30T14:30:00.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 181154 11.22 MB/sec execute 285 sec latency 56.234 ms 2024-03-30T14:30:00.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:59 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:00.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:59 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:00.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:59 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:00.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:59 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:00.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:29:59 smithi193 ceph-mon[52977]: pgmap v51: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 732 op/s 2024-03-30T14:30:00.934 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:00 smithi031 ceph-mon[67456]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:30:00.935 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:00 smithi031 ceph-mon[67456]: mgrmap e43: smithi031.lkrchi(active, since 92s), standbys: smithi193.ikzlnb 2024-03-30T14:30:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 181467 11.21 MB/sec execute 286 sec latency 40.276 ms 2024-03-30T14:30:01.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:00 smithi193 ceph-mon[52977]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:30:01.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:00 smithi193 ceph-mon[52977]: mgrmap e43: smithi031.lkrchi(active, since 92s), standbys: smithi193.ikzlnb 2024-03-30T14:30:01.980 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:01 smithi031 ceph-mon[67456]: pgmap v52: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-03-30T14:30:02.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:01 smithi193 ceph-mon[52977]: pgmap v52: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-03-30T14:30:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 181783 11.20 MB/sec execute 287 sec latency 66.676 ms 2024-03-30T14:30:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 182176 11.21 MB/sec execute 288 sec latency 33.126 ms 2024-03-30T14:30:03.754 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:03 smithi031 ceph-mon[67456]: pgmap v53: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.6 MiB/s wr, 652 op/s 2024-03-30T14:30:04.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:03 smithi193 ceph-mon[52977]: pgmap v53: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.6 MiB/s wr, 652 op/s 2024-03-30T14:30:04.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 182627 11.20 MB/sec execute 289 sec latency 32.217 ms 2024-03-30T14:30:05.138 INFO:teuthology.orchestra.run.smithi031.stdout: 5 182997 11.19 MB/sec execute 290 sec latency 40.443 ms 2024-03-30T14:30:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 183270 11.18 MB/sec execute 291 sec latency 67.741 ms 2024-03-30T14:30:06.309 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.309 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.309 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:06.309 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:30:06.309 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: pgmap v54: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: pgmap v55: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 550 op/s 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all crash 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi031"}]: dispatch 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi031"}]': finished 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi193"}]: dispatch 2024-03-30T14:30:06.310 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi193"}]': finished 2024-03-30T14:30:06.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T14:30:06.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T14:30:06.311 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:06 smithi031 ceph-mon[67456]: Upgrade: osd.0 is safe to restart 2024-03-30T14:30:06.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: pgmap v54: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: pgmap v55: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 550 op/s 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all crash 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:06.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi031"}]: dispatch 2024-03-30T14:30:06.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi031"}]': finished 2024-03-30T14:30:06.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi193"}]: dispatch 2024-03-30T14:30:06.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi193"}]': finished 2024-03-30T14:30:06.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T14:30:06.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T14:30:06.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:06 smithi193 ceph-mon[52977]: Upgrade: osd.0 is safe to restart 2024-03-30T14:30:07.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 183622 11.17 MB/sec execute 292 sec latency 80.393 ms 2024-03-30T14:30:08.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 184030 11.17 MB/sec execute 293 sec latency 41.230 ms 2024-03-30T14:30:08.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:08 smithi031 ceph-mon[67456]: pgmap v56: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 550 op/s 2024-03-30T14:30:08.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:08 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:08.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:08 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:30:08.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:08 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:08.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:08 smithi193 ceph-mon[52977]: pgmap v56: 97 pgs: 97 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 550 op/s 2024-03-30T14:30:08.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:08 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:08.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:08 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:30:08.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:08 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:09.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 184479 11.19 MB/sec execute 294 sec latency 39.829 ms 2024-03-30T14:30:09.415 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:09 smithi031 ceph-mon[67456]: Upgrade: Updating osd.0 2024-03-30T14:30:09.415 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:09 smithi031 ceph-mon[67456]: Deploying daemon osd.0 on smithi031 2024-03-30T14:30:09.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:09 smithi193 ceph-mon[52977]: Upgrade: Updating osd.0 2024-03-30T14:30:09.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:09 smithi193 ceph-mon[52977]: Deploying daemon osd.0 on smithi031 2024-03-30T14:30:10.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 184839 11.17 MB/sec execute 295 sec latency 51.005 ms 2024-03-30T14:30:10.437 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:10 smithi031 ceph-mon[67456]: pgmap v57: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 533 op/s 2024-03-30T14:30:10.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:10 smithi193 ceph-mon[52977]: pgmap v57: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 533 op/s 2024-03-30T14:30:11.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 185282 11.17 MB/sec execute 296 sec latency 32.656 ms 2024-03-30T14:30:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 185720 11.19 MB/sec execute 297 sec latency 44.372 ms 2024-03-30T14:30:12.457 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:12 smithi031 ceph-mon[67456]: pgmap v58: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 533 op/s 2024-03-30T14:30:12.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:12 smithi193 ceph-mon[52977]: pgmap v58: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 533 op/s 2024-03-30T14:30:13.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 186134 11.18 MB/sec execute 298 sec latency 41.883 ms 2024-03-30T14:30:13.217 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:13 smithi031 systemd[1]: Stopping Ceph osd.0 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:30:13.471 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:13.471 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:13 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[34948]: 2024-03-30T14:30:13.430+0000 7ff56afb2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:30:13.471 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:13 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[34948]: 2024-03-30T14:30:13.430+0000 7ff56afb2700 -1 osd.0 73 *** Got signal Terminated *** 2024-03-30T14:30:13.471 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:13 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[34948]: 2024-03-30T14:30:13.430+0000 7ff56afb2700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:30:13.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:14.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 186572 11.17 MB/sec execute 299 sec latency 40.716 ms 2024-03-30T14:30:14.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:14 smithi193 ceph-mon[52977]: pgmap v59: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-03-30T14:30:14.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:14 smithi193 ceph-mon[52977]: osd.0 marked itself down and dead 2024-03-30T14:30:14.609 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:14 smithi031 ceph-mon[67456]: pgmap v59: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-03-30T14:30:14.610 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:14 smithi031 ceph-mon[67456]: osd.0 marked itself down and dead 2024-03-30T14:30:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 186615 11.14 MB/sec execute 300 sec latency 901.218 ms 2024-03-30T14:30:15.162 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:14 smithi031 podman[72921]: 2024-03-30 14:30:14.877962051 +0000 UTC m=+1.593114610 container died 0bc8ae96f5b8ecb76137103affacf8bfcd0e275f2088fb34b3f8c4d5a0a1e8fa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, architecture=x86_64, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.buildah.version=1.19.8, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git) 2024-03-30T14:30:15.413 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:15 smithi031 podman[72921]: 2024-03-30 14:30:15.266526484 +0000 UTC m=+1.981679043 container cleanup 0bc8ae96f5b8ecb76137103affacf8bfcd0e275f2088fb34b3f8c4d5a0a1e8fa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, release=754, vcs-type=git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-03-30T14:30:15.413 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:15 smithi031 bash[72921]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T14:30:15.413 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:15 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:30:15.413 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:15 smithi031 ceph-mon[67456]: failed to encode map e74 with expected crc 2024-03-30T14:30:15.414 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:15 smithi031 ceph-mon[67456]: osdmap e74: 8 total, 7 up, 8 in 2024-03-30T14:30:15.414 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:15 smithi031 ceph-mon[67456]: failed to encode map e74 with expected crc 2024-03-30T14:30:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:15 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:30:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:15 smithi193 ceph-mon[52977]: failed to encode map e74 with expected crc 2024-03-30T14:30:15.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:15 smithi193 ceph-mon[52977]: osdmap e74: 8 total, 7 up, 8 in 2024-03-30T14:30:15.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:15 smithi193 ceph-mon[52977]: failed to encode map e74 with expected crc 2024-03-30T14:30:15.664 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:15 smithi031 podman[72937]: 2024-03-30 14:30:15.636973911 +0000 UTC m=+0.749207116 container remove 0bc8ae96f5b8ecb76137103affacf8bfcd0e275f2088fb34b3f8c4d5a0a1e8fa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, RELEASE=HEAD, ceph=True, com.redhat.component=centos-stream-container, vcs-type=git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, name=centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream) 2024-03-30T14:30:16.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 186944 11.14 MB/sec execute 301 sec latency 1070.495 ms 2024-03-30T14:30:16.170 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:16 smithi031 podman[73020]: 2024-03-30 14:30:15.945568039 +0000 UTC m=+0.021295290 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:30:16.421 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:16 smithi031 podman[73020]: 2024-03-30 14:30:16.237646675 +0000 UTC m=+0.313373901 container create 7dfe0f889502a0107336648629610f1600d8a3a11a901ab6df563e7c3ea94390 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc) 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: pgmap v61: 97 pgs: 13 stale+active+clean, 84 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: osdmap e75: 8 total, 7 up, 8 in 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.421 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.422 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.422 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:16 smithi031 ceph-mon[67456]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: pgmap v61: 97 pgs: 13 stale+active+clean, 84 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-03-30T14:30:16.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: osdmap e75: 8 total, 7 up, 8 in 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:16 smithi193 ceph-mon[52977]: failed to encode map e75 with expected crc 2024-03-30T14:30:16.949 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:16 smithi031 podman[73020]: 2024-03-30 14:30:16.925434092 +0000 UTC m=+1.001161336 container init 7dfe0f889502a0107336648629610f1600d8a3a11a901ab6df563e7c3ea94390 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True) 2024-03-30T14:30:16.949 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:16 smithi031 podman[73020]: 2024-03-30 14:30:16.932379504 +0000 UTC m=+1.008106734 container start 7dfe0f889502a0107336648629610f1600d8a3a11a901ab6df563e7c3ea94390 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS) 2024-03-30T14:30:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 187364 11.14 MB/sec execute 302 sec latency 45.338 ms 2024-03-30T14:30:17.201 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:17 smithi031 podman[73020]: 2024-03-30 14:30:17.080625276 +0000 UTC m=+1.156352511 container attach 7dfe0f889502a0107336648629610f1600d8a3a11a901ab6df563e7c3ea94390 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-30T14:30:17.452 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:17 smithi031 podman[73044]: 2024-03-30 14:30:17.399453204 +0000 UTC m=+0.021891524 container died 7dfe0f889502a0107336648629610f1600d8a3a11a901ab6df563e7c3ea94390 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-30T14:30:18.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 187638 11.12 MB/sec execute 303 sec latency 68.325 ms 2024-03-30T14:30:18.504 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:18 smithi031 podman[73044]: 2024-03-30 14:30:18.384185497 +0000 UTC m=+1.006623806 container remove 7dfe0f889502a0107336648629610f1600d8a3a11a901ab6df563e7c3ea94390 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-30T14:30:18.504 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:18 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.0.service: Deactivated successfully. 2024-03-30T14:30:18.504 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:18 smithi031 systemd[1]: Stopped Ceph osd.0 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:30:18.504 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:18 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.0.service: Consumed 3min 23.450s CPU time. 2024-03-30T14:30:18.504 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:18 smithi031 ceph-mon[67456]: pgmap v63: 97 pgs: 13 stale+active+clean, 84 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.8 MiB/s wr, 385 op/s 2024-03-30T14:30:18.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:18 smithi193 ceph-mon[52977]: pgmap v63: 97 pgs: 13 stale+active+clean, 84 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.8 MiB/s wr, 385 op/s 2024-03-30T14:30:18.755 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:18 smithi031 systemd[1]: Starting Ceph osd.0 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:30:19.008 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:18 smithi031 podman[73141]: 2024-03-30 14:30:18.857518812 +0000 UTC m=+0.024397112 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:30:19.008 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:18 smithi031 podman[73141]: 2024-03-30 14:30:18.977309803 +0000 UTC m=+0.144188082 container create f8b87c3a51e2888730c58af98f238b62376c7b1afbcdcc654bd82d6063c67dbd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-30T14:30:19.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 187869 11.11 MB/sec execute 304 sec latency 63.537 ms 2024-03-30T14:30:19.511 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 203/1590 objects degraded (12.767%), 29 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:19.511 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 podman[73141]: 2024-03-30 14:30:19.264996761 +0000 UTC m=+0.431875044 container init f8b87c3a51e2888730c58af98f238b62376c7b1afbcdcc654bd82d6063c67dbd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-30T14:30:19.511 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 podman[73141]: 2024-03-30 14:30:19.276531073 +0000 UTC m=+0.443409358 container start f8b87c3a51e2888730c58af98f238b62376c7b1afbcdcc654bd82d6063c67dbd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-30T14:30:19.511 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 podman[73141]: 2024-03-30 14:30:19.464762933 +0000 UTC m=+0.631641216 container attach f8b87c3a51e2888730c58af98f238b62376c7b1afbcdcc654bd82d6063c67dbd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-30T14:30:19.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:19 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 203/1590 objects degraded (12.767%), 29 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate[73155]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate[73155]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 bash[73141]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 bash[73141]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 bash[73141]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 bash[73141]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 bash[73141]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 bash[73141]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 bash[73141]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate[73155]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T14:30:20.030 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate[73155]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T14:30:20.031 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate[73155]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-30T14:30:20.031 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate[73155]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T14:30:20.031 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate[73155]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-30T14:30:20.031 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:19 smithi031 podman[73141]: 2024-03-30 14:30:19.865403056 +0000 UTC m=+1.032281351 container died f8b87c3a51e2888730c58af98f238b62376c7b1afbcdcc654bd82d6063c67dbd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:30:20.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 188083 11.10 MB/sec execute 305 sec latency 49.002 ms 2024-03-30T14:30:20.415 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:30:20.547 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:20 smithi031 ceph-mon[67456]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.6 MiB/s wr, 660 op/s; 203/1590 objects degraded (12.767%) 2024-03-30T14:30:20.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:20 smithi193 ceph-mon[52977]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.6 MiB/s wr, 660 op/s; 203/1590 objects degraded (12.767%) 2024-03-30T14:30:21.053 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:20 smithi031 podman[73282]: 2024-03-30 14:30:20.83883771 +0000 UTC m=+0.967505397 container remove f8b87c3a51e2888730c58af98f238b62376c7b1afbcdcc654bd82d6063c67dbd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T14:30:21.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 188334 11.08 MB/sec execute 306 sec latency 97.874 ms 2024-03-30T14:30:21.314 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:21 smithi031 podman[73390]: 2024-03-30 14:30:21.007198165 +0000 UTC m=+0.028665392 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:30:21.314 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:21 smithi031 podman[73390]: 2024-03-30 14:30:21.148782192 +0000 UTC m=+0.170249408 container create c1ab3320f0df33a978331cffae185e9e78ce1f0a49c8e14ef5e38d209003a7e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux ) 2024-03-30T14:30:21.379 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:30:21.379 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (4m) 23s ago 13m 16.4M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:30:21.379 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (38s) 23s ago 13m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:30:21.379 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (32s) 27s ago 12m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:30:21.379 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 23s ago 13m - - 2024-03-30T14:30:21.379 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (7m) 23s ago 7m 40.6M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:30:21.379 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (7m) 27s ago 7m 14.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (3m) 23s ago 14m 500M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (2m) 27s ago 12m 449M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (2m) 23s ago 15m 43.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (94s) 27s ago 12m 36.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (7m) 23s ago 7m 268M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (7m) 27s ago 7m 82.5M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (4m) 23s ago 13m 13.6M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (3m) 27s ago 12m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (12m) 23s ago 12m 371M 1937M 17.2.0 e1d6a67b021e 0bc8ae96f5b8 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (11m) 23s ago 11m 558M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (11m) 23s ago 11m 480M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (10m) 23s ago 10m 552M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (10m) 27s ago 10m 546M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:30:21.380 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (9m) 27s ago 9m 402M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:30:21.381 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (9m) 27s ago 9m 357M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:30:21.381 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (9m) 27s ago 9m 453M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:30:21.381 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (2m) 23s ago 12m 59.2M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:30:21.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:21 smithi031 ceph-mon[67456]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:21.816 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:21 smithi031 podman[73390]: 2024-03-30 14:30:21.603579258 +0000 UTC m=+0.625046489 container init c1ab3320f0df33a978331cffae185e9e78ce1f0a49c8e14ef5e38d209003a7e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:30:21.816 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:21 smithi031 podman[73390]: 2024-03-30 14:30:21.616516758 +0000 UTC m=+0.637983971 container start c1ab3320f0df33a978331cffae185e9e78ce1f0a49c8e14ef5e38d209003a7e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2) 2024-03-30T14:30:21.816 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:21 smithi031 ceph-osd[73478]: -- 172.21.15.31:0/1149645658 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648cc57d680 con 0x5648cc546c00 2024-03-30T14:30:21.816 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:21 smithi031 bash[73390]: c1ab3320f0df33a978331cffae185e9e78ce1f0a49c8e14ef5e38d209003a7e1 2024-03-30T14:30:21.817 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:21 smithi031 systemd[1]: Started Ceph osd.0 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:30:21.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:21 smithi193 ceph-mon[52977]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:22.094 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:30:22.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:30:22.094 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:30:22.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:22.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:30:22.094 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:30:22.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 4 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:30:22.095 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:30:22.139 INFO:teuthology.orchestra.run.smithi031.stdout: 5 188681 11.08 MB/sec execute 307 sec latency 60.014 ms 2024-03-30T14:30:22.319 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[73454]: 2024-03-30T14:30:22.200+0000 7f4d9b27a7c0 -1 Falling back to public interface 2024-03-30T14:30:22.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-mon[67456]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:22.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-mon[67456]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.6 MiB/s wr, 660 op/s; 203/1590 objects degraded (12.767%) 2024-03-30T14:30:22.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-mon[67456]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:22.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:22.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:22.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:22.573 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:22 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/411503724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:22.645 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:30:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:30:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:30:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:30:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "6/23 daemons upgraded", 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:30:22.646 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:30:22.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:22 smithi193 ceph-mon[52977]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:22.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:22 smithi193 ceph-mon[52977]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.6 MiB/s wr, 660 op/s; 203/1590 objects degraded (12.767%) 2024-03-30T14:30:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:22 smithi193 ceph-mon[52977]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:22 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:22 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/411503724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 188895 11.06 MB/sec execute 308 sec latency 45.534 ms 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 203/1590 objects degraded (12.767%), 29 pgs degraded 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: osd.0 (root=default,host=smithi031) is down 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 203/1590 objects degraded (12.767%), 29 pgs degraded 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-03-30T14:30:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.5 is active+undersized+degraded, acting [7,4] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.19 is active+undersized+degraded, acting [3,6] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1d is active+undersized+degraded, acting [7,6] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1e is active+undersized+degraded, acting [3,5] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-03-30T14:30:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-03-30T14:30:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-03-30T14:30:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 189216 11.06 MB/sec execute 309 sec latency 51.261 ms 2024-03-30T14:30:24.377 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:24 smithi031 ceph-mon[67456]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:24.377 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:24 smithi031 ceph-mon[67456]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.5 MiB/s wr, 247 op/s; 203/1590 objects degraded (12.767%) 2024-03-30T14:30:24.377 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:24 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2096062571' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:30:24.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:24 smithi193 ceph-mon[52977]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:24.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:24 smithi193 ceph-mon[52977]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.5 MiB/s wr, 247 op/s; 203/1590 objects degraded (12.767%) 2024-03-30T14:30:24.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:24 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2096062571' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:30:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 189597 11.05 MB/sec execute 310 sec latency 61.966 ms 2024-03-30T14:30:25.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:25 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 219/1629 objects degraded (13.444%), 29 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:25.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:25 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 219/1629 objects degraded (13.444%), 29 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 190021 11.04 MB/sec execute 311 sec latency 37.313 ms 2024-03-30T14:30:26.652 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:26 smithi031 ceph-mon[67456]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 219/1629 objects degraded (13.444%) 2024-03-30T14:30:26.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:26 smithi193 ceph-mon[52977]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 219/1629 objects degraded (13.444%) 2024-03-30T14:30:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 190413 11.05 MB/sec execute 312 sec latency 40.576 ms 2024-03-30T14:30:28.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 190828 11.05 MB/sec execute 313 sec latency 51.077 ms 2024-03-30T14:30:28.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:28 smithi031 ceph-mon[67456]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.6 MiB/s wr, 346 op/s; 219/1629 objects degraded (13.444%) 2024-03-30T14:30:28.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:28.711 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:29.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:28 smithi193 ceph-mon[52977]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.6 MiB/s wr, 346 op/s; 219/1629 objects degraded (13.444%) 2024-03-30T14:30:29.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:29.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:29.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 191177 11.04 MB/sec execute 314 sec latency 56.077 ms 2024-03-30T14:30:29.737 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:29 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[73454]: 2024-03-30T14:30:29.566+0000 7f4d9b27a7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-30T14:30:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 191487 11.03 MB/sec execute 315 sec latency 48.410 ms 2024-03-30T14:30:30.774 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:30 smithi031 ceph-mon[67456]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 580 op/s; 214/1656 objects degraded (12.923%) 2024-03-30T14:30:31.033 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:30 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[73454]: 2024-03-30T14:30:30.933+0000 7f4d9b27a7c0 -1 osd.0 73 log_to_monitors true 2024-03-30T14:30:31.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:30 smithi193 ceph-mon[52977]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 580 op/s; 214/1656 objects degraded (12.923%) 2024-03-30T14:30:31.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 191797 11.02 MB/sec execute 316 sec latency 47.907 ms 2024-03-30T14:30:31.790 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:31.790 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:31.790 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:31 smithi031 ceph-mon[67456]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T14:30:31.790 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:31 smithi031 ceph-mon[67456]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 397 op/s; 214/1656 objects degraded (12.923%) 2024-03-30T14:30:32.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:32.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:32.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:31 smithi193 ceph-mon[52977]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T14:30:32.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:31 smithi193 ceph-mon[52977]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 397 op/s; 214/1656 objects degraded (12.923%) 2024-03-30T14:30:32.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 192095 11.01 MB/sec execute 317 sec latency 199.051 ms 2024-03-30T14:30:32.313 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[73454]: 2024-03-30T14:30:32.217+0000 7f4d886ca700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:30:32.827 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T14:30:32.827 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: failed to encode map e76 with expected crc 2024-03-30T14:30:32.827 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: osdmap e76: 8 total, 7 up, 8 in 2024-03-30T14:30:32.827 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:30:32.827 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: failed to encode map e76 with expected crc 2024-03-30T14:30:32.827 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: failed to encode map e76 with expected crc 2024-03-30T14:30:32.828 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' 2024-03-30T14:30:32.828 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:32.828 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:33.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T14:30:33.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: failed to encode map e76 with expected crc 2024-03-30T14:30:33.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: osdmap e76: 8 total, 7 up, 8 in 2024-03-30T14:30:33.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:30:33.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: failed to encode map e76 with expected crc 2024-03-30T14:30:33.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: failed to encode map e76 with expected crc 2024-03-30T14:30:33.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: from='osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621]' entity='osd.0' 2024-03-30T14:30:33.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:33.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:33.140 INFO:teuthology.orchestra.run.smithi031.stdout: 5 192456 11.01 MB/sec execute 318 sec latency 48.237 ms 2024-03-30T14:30:33.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:33 smithi031 ceph-mon[67456]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 476 op/s; 214/1656 objects degraded (12.923%) 2024-03-30T14:30:33.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:33 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:30:33.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:33 smithi031 ceph-mon[67456]: failed to encode map e77 with expected crc 2024-03-30T14:30:33.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:33 smithi031 ceph-mon[67456]: osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621] boot 2024-03-30T14:30:33.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:33 smithi031 ceph-mon[67456]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T14:30:33.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:30:33.845 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:33 smithi031 ceph-mon[67456]: failed to encode map e77 with expected crc 2024-03-30T14:30:34.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:33 smithi193 ceph-mon[52977]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 476 op/s; 214/1656 objects degraded (12.923%) 2024-03-30T14:30:34.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:33 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:30:34.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:33 smithi193 ceph-mon[52977]: failed to encode map e77 with expected crc 2024-03-30T14:30:34.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:33 smithi193 ceph-mon[52977]: osd.0 [v2:172.21.15.31:6802/670015621,v1:172.21.15.31:6803/670015621] boot 2024-03-30T14:30:34.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:33 smithi193 ceph-mon[52977]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T14:30:34.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:30:34.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:33 smithi193 ceph-mon[52977]: failed to encode map e77 with expected crc 2024-03-30T14:30:34.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 192581 10.98 MB/sec execute 319 sec latency 926.970 ms 2024-03-30T14:30:35.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 192645 10.95 MB/sec execute 320 sec latency 1926.995 ms 2024-03-30T14:30:35.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.312 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.313 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:35 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 214/1656 objects degraded (12.923%), 29 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:35.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T14:30:35.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: failed to encode map e78 with expected crc 2024-03-30T14:30:35.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:35 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 214/1656 objects degraded (12.923%), 29 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 192970 10.95 MB/sec execute 321 sec latency 2112.583 ms 2024-03-30T14:30:36.321 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:36 smithi031 ceph-mon[67456]: pgmap v75: 97 pgs: 23 peering, 8 active+undersized+degraded, 5 active+undersized, 61 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.5 MiB/s wr, 360 op/s; 94/1728 objects degraded (5.440%) 2024-03-30T14:30:36.321 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:36 smithi031 ceph-mon[67456]: failed to encode map e79 with expected crc 2024-03-30T14:30:36.321 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:36 smithi031 ceph-mon[67456]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T14:30:36.321 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:36 smithi031 ceph-mon[67456]: failed to encode map e79 with expected crc 2024-03-30T14:30:36.321 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:36 smithi031 ceph-mon[67456]: failed to encode map e79 with expected crc 2024-03-30T14:30:36.321 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:36 smithi031 ceph-mon[67456]: failed to encode map e79 with expected crc 2024-03-30T14:30:36.484 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:36 smithi193 ceph-mon[52977]: pgmap v75: 97 pgs: 23 peering, 8 active+undersized+degraded, 5 active+undersized, 61 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.5 MiB/s wr, 360 op/s; 94/1728 objects degraded (5.440%) 2024-03-30T14:30:36.484 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:36 smithi193 ceph-mon[52977]: failed to encode map e79 with expected crc 2024-03-30T14:30:36.484 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:36 smithi193 ceph-mon[52977]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T14:30:36.484 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:36 smithi193 ceph-mon[52977]: failed to encode map e79 with expected crc 2024-03-30T14:30:36.484 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:36 smithi193 ceph-mon[52977]: failed to encode map e79 with expected crc 2024-03-30T14:30:36.484 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:36 smithi193 ceph-mon[52977]: failed to encode map e79 with expected crc 2024-03-30T14:30:37.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 193476 10.96 MB/sec execute 322 sec latency 30.898 ms 2024-03-30T14:30:37.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:37 smithi031 ceph-mon[67456]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:37 smithi031 ceph-mon[67456]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T14:30:37.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:37 smithi031 ceph-mon[67456]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:37 smithi031 ceph-mon[67456]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:37 smithi031 ceph-mon[67456]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:37 smithi031 ceph-mon[67456]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:37 smithi193 ceph-mon[52977]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:37 smithi193 ceph-mon[52977]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T14:30:37.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:37 smithi193 ceph-mon[52977]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:37 smithi193 ceph-mon[52977]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:37 smithi193 ceph-mon[52977]: failed to encode map e80 with expected crc 2024-03-30T14:30:37.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:37 smithi193 ceph-mon[52977]: failed to encode map e80 with expected crc 2024-03-30T14:30:38.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 193851 10.95 MB/sec execute 323 sec latency 463.291 ms 2024-03-30T14:30:38.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: pgmap v78: 97 pgs: 23 peering, 4 active+undersized+degraded, 70 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.4 MiB/s wr, 554 op/s; 61/1731 objects degraded (3.524%); 0 B/s, 9 objects/s recovering 2024-03-30T14:30:38.345 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: pgmap v80: 97 pgs: 23 peering, 4 active+undersized+degraded, 70 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 247 KiB/s wr, 18 op/s; 61/1731 objects degraded (3.524%); 0 B/s, 12 objects/s recovering 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: pgmap v81: 97 pgs: 23 peering, 4 active+undersized+degraded, 70 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 357 KiB/s wr, 26 op/s; 61/1731 objects degraded (3.524%); 0 B/s, 17 objects/s recovering 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:38.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:38.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:38.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:38.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:38.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:38.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:38 smithi031 ceph-mon[67456]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-30T14:30:38.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: pgmap v78: 97 pgs: 23 peering, 4 active+undersized+degraded, 70 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.4 MiB/s wr, 554 op/s; 61/1731 objects degraded (3.524%); 0 B/s, 9 objects/s recovering 2024-03-30T14:30:38.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: failed to encode map e81 with expected crc 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: pgmap v80: 97 pgs: 23 peering, 4 active+undersized+degraded, 70 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 247 KiB/s wr, 18 op/s; 61/1731 objects degraded (3.524%); 0 B/s, 12 objects/s recovering 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: pgmap v81: 97 pgs: 23 peering, 4 active+undersized+degraded, 70 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 357 KiB/s wr, 26 op/s; 61/1731 objects degraded (3.524%); 0 B/s, 17 objects/s recovering 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:38.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:38.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:38 smithi193 ceph-mon[52977]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-30T14:30:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 194243 10.95 MB/sec execute 324 sec latency 577.769 ms 2024-03-30T14:30:39.371 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:39 smithi031 ceph-mon[67456]: failed to encode map e82 with expected crc 2024-03-30T14:30:39.371 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:39 smithi031 ceph-mon[67456]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T14:30:39.371 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:39 smithi031 ceph-mon[67456]: failed to encode map e82 with expected crc 2024-03-30T14:30:39.371 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:39 smithi031 ceph-mon[67456]: failed to encode map e82 with expected crc 2024-03-30T14:30:39.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:39 smithi193 ceph-mon[52977]: failed to encode map e82 with expected crc 2024-03-30T14:30:39.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:39 smithi193 ceph-mon[52977]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T14:30:39.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:39 smithi193 ceph-mon[52977]: failed to encode map e82 with expected crc 2024-03-30T14:30:39.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:39 smithi193 ceph-mon[52977]: failed to encode map e82 with expected crc 2024-03-30T14:30:40.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 194597 10.95 MB/sec execute 325 sec latency 588.632 ms 2024-03-30T14:30:40.407 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.407 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T14:30:40.407 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.407 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.407 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.408 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.408 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.408 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.408 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: pgmap v84: 97 pgs: 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 18 KiB/s rd, 9.3 MiB/s wr, 1.01k op/s; 118/1695 objects degraded (6.962%); 4.8 MiB/s, 50 objects/s recovering 2024-03-30T14:30:40.408 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:40 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 61/1731 objects degraded (3.524%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: failed to encode map e83 with expected crc 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: pgmap v84: 97 pgs: 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 18 KiB/s rd, 9.3 MiB/s wr, 1.01k op/s; 118/1695 objects degraded (6.962%); 4.8 MiB/s, 50 objects/s recovering 2024-03-30T14:30:40.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:40 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 61/1731 objects degraded (3.524%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:41.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 194985 10.96 MB/sec execute 326 sec latency 727.201 ms 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.430 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:41 smithi031 ceph-mon[67456]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T14:30:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: failed to encode map e84 with expected crc 2024-03-30T14:30:41.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:41 smithi193 ceph-mon[52977]: failed to encode map e84 with expected crc 2024-03-30T14:30:42.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 195215 10.95 MB/sec execute 327 sec latency 884.065 ms 2024-03-30T14:30:42.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:42 smithi031 ceph-mon[67456]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:42 smithi031 ceph-mon[67456]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T14:30:42.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:42 smithi031 ceph-mon[67456]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:42 smithi031 ceph-mon[67456]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:42 smithi031 ceph-mon[67456]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:42 smithi031 ceph-mon[67456]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.507 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:42 smithi031 ceph-mon[67456]: pgmap v87: 97 pgs: 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.0 MiB/s wr, 543 op/s; 118/1695 objects degraded (6.962%); 2.6 MiB/s, 27 objects/s recovering 2024-03-30T14:30:42.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:42 smithi193 ceph-mon[52977]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:42 smithi193 ceph-mon[52977]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T14:30:42.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:42 smithi193 ceph-mon[52977]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:42 smithi193 ceph-mon[52977]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:42 smithi193 ceph-mon[52977]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:42 smithi193 ceph-mon[52977]: failed to encode map e85 with expected crc 2024-03-30T14:30:42.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:42 smithi193 ceph-mon[52977]: pgmap v87: 97 pgs: 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.0 MiB/s wr, 543 op/s; 118/1695 objects degraded (6.962%); 2.6 MiB/s, 27 objects/s recovering 2024-03-30T14:30:43.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 195556 10.93 MB/sec execute 328 sec latency 1033.483 ms 2024-03-30T14:30:43.265 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.265 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T14:30:43.265 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.265 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.265 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.266 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.266 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.266 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:43.266 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:43.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: failed to encode map e86 with expected crc 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:43.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 195842 10.91 MB/sec execute 329 sec latency 56.704 ms 2024-03-30T14:30:44.285 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:44 smithi031 ceph-mon[67456]: pgmap v89: 97 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 peering, 92 active+clean; 352 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s wr, 320 op/s; 45/1698 objects degraded (2.650%); 0 B/s, 16 objects/s recovering 2024-03-30T14:30:44.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:44 smithi193 ceph-mon[52977]: pgmap v89: 97 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 peering, 92 active+clean; 352 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 2.6 MiB/s wr, 320 op/s; 45/1698 objects degraded (2.650%); 0 B/s, 16 objects/s recovering 2024-03-30T14:30:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 196163 10.91 MB/sec execute 330 sec latency 35.927 ms 2024-03-30T14:30:45.300 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:45 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 45/1698 objects degraded (2.650%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:45.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:45 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 45/1698 objects degraded (2.650%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T14:30:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 196467 10.91 MB/sec execute 331 sec latency 48.872 ms 2024-03-30T14:30:46.354 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:46 smithi031 ceph-mon[67456]: pgmap v90: 97 pgs: 2 peering, 95 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 419 op/s; 0 B/s, 18 objects/s recovering 2024-03-30T14:30:46.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:46 smithi193 ceph-mon[52977]: pgmap v90: 97 pgs: 2 peering, 95 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 419 op/s; 0 B/s, 18 objects/s recovering 2024-03-30T14:30:47.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 196840 10.91 MB/sec execute 332 sec latency 51.763 ms 2024-03-30T14:30:47.373 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:47 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/1698 objects degraded (2.650%), 2 pgs degraded) 2024-03-30T14:30:47.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:47 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/1698 objects degraded (2.650%), 2 pgs degraded) 2024-03-30T14:30:48.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 197180 10.89 MB/sec execute 333 sec latency 52.054 ms 2024-03-30T14:30:48.394 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:48 smithi031 ceph-mon[67456]: pgmap v91: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.6 MiB/s wr, 382 op/s; 0 B/s, 18 objects/s recovering 2024-03-30T14:30:48.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:48 smithi193 ceph-mon[52977]: pgmap v91: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.6 MiB/s wr, 382 op/s; 0 B/s, 18 objects/s recovering 2024-03-30T14:30:49.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 197459 10.88 MB/sec execute 334 sec latency 40.684 ms 2024-03-30T14:30:50.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 197718 10.86 MB/sec execute 335 sec latency 58.280 ms 2024-03-30T14:30:50.471 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:50 smithi031 ceph-mon[67456]: pgmap v92: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.3 MiB/s wr, 614 op/s; 0 B/s, 16 objects/s recovering 2024-03-30T14:30:50.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:50 smithi193 ceph-mon[52977]: pgmap v92: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.3 MiB/s wr, 614 op/s; 0 B/s, 16 objects/s recovering 2024-03-30T14:30:51.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 198021 10.87 MB/sec execute 336 sec latency 50.788 ms 2024-03-30T14:30:52.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 198323 10.86 MB/sec execute 337 sec latency 65.985 ms 2024-03-30T14:30:52.518 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:52 smithi031 ceph-mon[67456]: pgmap v93: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 494 op/s; 0 B/s, 12 objects/s recovering 2024-03-30T14:30:52.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:52 smithi193 ceph-mon[52977]: pgmap v93: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 494 op/s; 0 B/s, 12 objects/s recovering 2024-03-30T14:30:53.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 198828 10.88 MB/sec execute 338 sec latency 41.670 ms 2024-03-30T14:30:53.529 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:53.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:53 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:53.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:53 smithi031 ceph-mon[67456]: Upgrade: osd.1 is safe to restart 2024-03-30T14:30:53.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:53.832 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:53.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:53 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T14:30:53.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:53 smithi193 ceph-mon[52977]: Upgrade: osd.1 is safe to restart 2024-03-30T14:30:53.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:53.976 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:30:54.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 199362 10.87 MB/sec execute 339 sec latency 23.864 ms 2024-03-30T14:30:54.789 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:54 smithi031 ceph-mon[67456]: pgmap v94: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.0 MiB/s wr, 345 op/s; 0 B/s, 5 objects/s recovering 2024-03-30T14:30:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:30:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (5m) 24s ago 14m 24.4M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:30:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (72s) 24s ago 14m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:30:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (66s) 61s ago 13m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:30:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 24s ago 13m - - 2024-03-30T14:30:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (8m) 24s ago 8m 40.6M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (8m) 61s ago 8m 14.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (4m) 24s ago 15m 502M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (3m) 61s ago 13m 449M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (2m) 24s ago 15m 47.3M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (2m) 61s ago 13m 36.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (7m) 24s ago 7m 269M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (7m) 61s ago 7m 82.5M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (4m) 24s ago 13m 13.5M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (4m) 61s ago 13m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (34s) 24s ago 12m 12.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (12m) 24s ago 12m 585M 1937M 17.2.0 e1d6a67b021e 5a6c177a1581 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (11m) 24s ago 11m 522M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (11m) 24s ago 11m 594M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:30:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (10m) 61s ago 10m 546M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:30:55.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (10m) 61s ago 10m 402M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:30:55.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (10m) 61s ago 10m 357M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:30:55.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (9m) 61s ago 9m 453M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:30:55.059 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (3m) 24s ago 12m 59.4M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:30:55.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:54 smithi193 ceph-mon[52977]: pgmap v94: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.0 MiB/s wr, 345 op/s; 0 B/s, 5 objects/s recovering 2024-03-30T14:30:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 199951 10.90 MB/sec execute 340 sec latency 28.800 ms 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:30:55.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 5 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:30:55.794 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:30:55.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:55 smithi031 ceph-mon[67456]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:55.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:55 smithi031 ceph-mon[67456]: from='client.44173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:55.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:55.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:30:55.795 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:56.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:55 smithi193 ceph-mon[52977]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:56.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:55 smithi193 ceph-mon[52977]: from='client.44173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:56.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:56.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:30:56.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:56.141 INFO:teuthology.orchestra.run.smithi031.stdout: 5 200478 10.91 MB/sec execute 341 sec latency 43.408 ms 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "7/23 daemons upgraded", 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:30:56.407 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:30:56.408 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:30:56.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:56 smithi031 ceph-mon[67456]: from='client.44177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:56.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:56 smithi031 ceph-mon[67456]: pgmap v95: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.5 MiB/s wr, 516 op/s; 0 B/s, 5 objects/s recovering 2024-03-30T14:30:56.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:56 smithi031 ceph-mon[67456]: Upgrade: Updating osd.1 2024-03-30T14:30:56.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:56 smithi031 ceph-mon[67456]: Deploying daemon osd.1 on smithi031 2024-03-30T14:30:56.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:56 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2583914157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:57.076 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:30:57.076 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:30:57.076 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:30:57.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:56 smithi193 ceph-mon[52977]: from='client.44177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:57.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:56 smithi193 ceph-mon[52977]: pgmap v95: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.5 MiB/s wr, 516 op/s; 0 B/s, 5 objects/s recovering 2024-03-30T14:30:57.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:56 smithi193 ceph-mon[52977]: Upgrade: Updating osd.1 2024-03-30T14:30:57.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:56 smithi193 ceph-mon[52977]: Deploying daemon osd.1 on smithi031 2024-03-30T14:30:57.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:56 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2583914157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:30:57.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 200906 10.89 MB/sec execute 342 sec latency 51.889 ms 2024-03-30T14:30:57.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:57 smithi031 ceph-mon[67456]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:57.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:57 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1568540355' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:30:57.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:57.806 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:58.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:57 smithi193 ceph-mon[52977]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:30:58.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:57 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1568540355' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:30:58.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:30:58.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 201250 10.89 MB/sec execute 343 sec latency 34.282 ms 2024-03-30T14:30:58.826 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:30:58 smithi031 ceph-mon[67456]: pgmap v96: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 439 op/s; 0 B/s, 1 objects/s recovering 2024-03-30T14:30:59.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:30:58 smithi193 ceph-mon[52977]: pgmap v96: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 439 op/s; 0 B/s, 1 objects/s recovering 2024-03-30T14:30:59.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 201657 10.91 MB/sec execute 344 sec latency 32.712 ms 2024-03-30T14:31:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 202103 10.92 MB/sec execute 345 sec latency 40.811 ms 2024-03-30T14:31:00.613 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:00 smithi031 systemd[1]: Stopping Ceph osd.1 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:31:00.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:00 smithi031 ceph-mon[67456]: pgmap v97: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 694 op/s 2024-03-30T14:31:01.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:00 smithi193 ceph-mon[52977]: pgmap v97: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 694 op/s 2024-03-30T14:31:01.119 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:01 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[38174]: 2024-03-30T14:31:01.033+0000 7fd48d989700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:31:01.119 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:01 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[38174]: 2024-03-30T14:31:01.033+0000 7fd48d989700 -1 osd.1 86 *** Got signal Terminated *** 2024-03-30T14:31:01.119 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:01 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[38174]: 2024-03-30T14:31:01.033+0000 7fd48d989700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:31:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 202502 10.91 MB/sec execute 346 sec latency 58.334 ms 2024-03-30T14:31:01.636 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:01 smithi031 ceph-mon[67456]: osd.1 marked itself down and dead 2024-03-30T14:31:02.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:01 smithi193 ceph-mon[52977]: osd.1 marked itself down and dead 2024-03-30T14:31:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 202783 10.89 MB/sec execute 347 sec latency 488.028 ms 2024-03-30T14:31:02.610 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:02 smithi031 podman[75587]: 2024-03-30 14:31:02.288919508 +0000 UTC m=+1.476769539 container died 5a6c177a15810fdfb31636a6fdaf1a84e590752939980a89bac4e9ece64fc463 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, release=754, GIT_CLEAN=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True) 2024-03-30T14:31:02.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:02 smithi031 ceph-mon[67456]: pgmap v98: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 507 op/s 2024-03-30T14:31:02.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:02 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:31:02.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:02 smithi031 ceph-mon[67456]: failed to encode map e87 with expected crc 2024-03-30T14:31:02.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:02 smithi031 ceph-mon[67456]: osdmap e87: 8 total, 7 up, 8 in 2024-03-30T14:31:02.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:02 smithi031 ceph-mon[67456]: failed to encode map e87 with expected crc 2024-03-30T14:31:02.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:02 smithi031 ceph-mon[67456]: failed to encode map e87 with expected crc 2024-03-30T14:31:03.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:02 smithi193 ceph-mon[52977]: pgmap v98: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 507 op/s 2024-03-30T14:31:03.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:02 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:31:03.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:02 smithi193 ceph-mon[52977]: failed to encode map e87 with expected crc 2024-03-30T14:31:03.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:02 smithi193 ceph-mon[52977]: osdmap e87: 8 total, 7 up, 8 in 2024-03-30T14:31:03.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:02 smithi193 ceph-mon[52977]: failed to encode map e87 with expected crc 2024-03-30T14:31:03.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:02 smithi193 ceph-mon[52977]: failed to encode map e87 with expected crc 2024-03-30T14:31:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 202917 10.86 MB/sec execute 348 sec latency 1017.750 ms 2024-03-30T14:31:03.376 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:03 smithi031 podman[75587]: 2024-03-30 14:31:03.164195325 +0000 UTC m=+2.352045363 container cleanup 5a6c177a15810fdfb31636a6fdaf1a84e590752939980a89bac4e9ece64fc463 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.expose-services=, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, ceph=True, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-type=git, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-03-30T14:31:03.376 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:03 smithi031 bash[75587]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T14:31:03.627 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:03 smithi031 podman[75634]: 2024-03-30 14:31:03.527896915 +0000 UTC m=+1.242366895 container remove 5a6c177a15810fdfb31636a6fdaf1a84e590752939980a89bac4e9ece64fc463 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.buildah.version=1.19.8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True) 2024-03-30T14:31:03.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: failed to encode map e88 with expected crc 2024-03-30T14:31:03.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: osdmap e88: 8 total, 7 up, 8 in 2024-03-30T14:31:03.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: failed to encode map e88 with expected crc 2024-03-30T14:31:03.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: failed to encode map e88 with expected crc 2024-03-30T14:31:03.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: failed to encode map e88 with expected crc 2024-03-30T14:31:03.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: failed to encode map e88 with expected crc 2024-03-30T14:31:03.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: failed to encode map e88 with expected crc 2024-03-30T14:31:03.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: failed to encode map e88 with expected crc 2024-03-30T14:31:03.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:03 smithi031 ceph-mon[67456]: pgmap v101: 97 pgs: 3 peering, 13 stale+active+clean, 81 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.1 MiB/s wr, 467 op/s 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: failed to encode map e88 with expected crc 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: osdmap e88: 8 total, 7 up, 8 in 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: failed to encode map e88 with expected crc 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: failed to encode map e88 with expected crc 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: failed to encode map e88 with expected crc 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: failed to encode map e88 with expected crc 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: failed to encode map e88 with expected crc 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: failed to encode map e88 with expected crc 2024-03-30T14:31:04.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:03 smithi193 ceph-mon[52977]: pgmap v101: 97 pgs: 3 peering, 13 stale+active+clean, 81 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.1 MiB/s wr, 467 op/s 2024-03-30T14:31:04.131 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:03 smithi031 podman[75719]: 2024-03-30 14:31:03.86814477 +0000 UTC m=+0.026477972 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:31:04.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 203368 10.87 MB/sec execute 349 sec latency 35.577 ms 2024-03-30T14:31:04.395 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:04 smithi031 podman[75719]: 2024-03-30 14:31:04.26323237 +0000 UTC m=+0.421565575 container create 6d3e6ab0f6868277dd4448592da3407b3504200d4fa4da42d8b9e336f3015a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-30T14:31:04.899 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:04 smithi031 podman[75719]: 2024-03-30 14:31:04.659678117 +0000 UTC m=+0.818011320 container init 6d3e6ab0f6868277dd4448592da3407b3504200d4fa4da42d8b9e336f3015a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:31:04.899 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:04 smithi031 podman[75719]: 2024-03-30 14:31:04.669076528 +0000 UTC m=+0.827409737 container start 6d3e6ab0f6868277dd4448592da3407b3504200d4fa4da42d8b9e336f3015a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-30T14:31:04.900 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:04 smithi031 podman[75719]: 2024-03-30 14:31:04.813871832 +0000 UTC m=+0.972205034 container attach 6d3e6ab0f6868277dd4448592da3407b3504200d4fa4da42d8b9e336f3015a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326) 2024-03-30T14:31:05.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 203875 10.89 MB/sec execute 350 sec latency 34.650 ms 2024-03-30T14:31:05.153 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:05 smithi031 podman[75719]: 2024-03-30 14:31:05.124243458 +0000 UTC m=+1.282576659 container died 6d3e6ab0f6868277dd4448592da3407b3504200d4fa4da42d8b9e336f3015a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-30T14:31:05.669 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:05 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 163/1659 objects degraded (9.825%), 22 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:05.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:05 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 163/1659 objects degraded (9.825%), 22 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 204273 10.88 MB/sec execute 351 sec latency 50.305 ms 2024-03-30T14:31:06.424 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:06 smithi031 podman[75744]: 2024-03-30 14:31:06.386952951 +0000 UTC m=+1.251470735 container remove 6d3e6ab0f6868277dd4448592da3407b3504200d4fa4da42d8b9e336f3015a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-03-30T14:31:06.424 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:06 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.1.service: Deactivated successfully. 2024-03-30T14:31:06.424 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:06 smithi031 systemd[1]: Stopped Ceph osd.1 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:31:06.424 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:06 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.1.service: Consumed 3min 58.884s CPU time. 2024-03-30T14:31:06.681 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:06 smithi031 ceph-mon[67456]: pgmap v102: 97 pgs: 22 active+undersized+degraded, 3 peering, 14 active+undersized, 58 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 722 op/s; 163/1659 objects degraded (9.825%) 2024-03-30T14:31:06.734 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:06 smithi193 ceph-mon[52977]: pgmap v102: 97 pgs: 22 active+undersized+degraded, 3 peering, 14 active+undersized, 58 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 722 op/s; 163/1659 objects degraded (9.825%) 2024-03-30T14:31:06.932 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:06 smithi031 systemd[1]: Starting Ceph osd.1 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:31:07.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 204615 10.87 MB/sec execute 352 sec latency 40.808 ms 2024-03-30T14:31:07.185 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:07 smithi031 podman[75840]: 2024-03-30 14:31:06.930999012 +0000 UTC m=+0.028603734 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:31:07.185 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:07 smithi031 podman[75840]: 2024-03-30 14:31:07.147064413 +0000 UTC m=+0.244669109 container create ab0117566734c09e47b299c7da2709dc7ebf0bf3cfe7ff861685063a5c0e1faf (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True) 2024-03-30T14:31:07.696 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:07 smithi031 podman[75840]: 2024-03-30 14:31:07.544544241 +0000 UTC m=+0.642148947 container init ab0117566734c09e47b299c7da2709dc7ebf0bf3cfe7ff861685063a5c0e1faf (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-03-30T14:31:07.696 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:07 smithi031 podman[75840]: 2024-03-30 14:31:07.55257444 +0000 UTC m=+0.650179144 container start ab0117566734c09e47b299c7da2709dc7ebf0bf3cfe7ff861685063a5c0e1faf (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-30T14:31:07.947 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:07 smithi031 podman[75840]: 2024-03-30 14:31:07.753501795 +0000 UTC m=+0.851106501 container attach ab0117566734c09e47b299c7da2709dc7ebf0bf3cfe7ff861685063a5c0e1faf (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-30T14:31:08.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 204993 10.87 MB/sec execute 353 sec latency 33.520 ms 2024-03-30T14:31:08.198 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate[75853]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate[75853]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 bash[75840]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 bash[75840]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 bash[75840]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 bash[75840]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 bash[75840]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 bash[75840]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 bash[75840]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate[75853]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate[75853]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-30T14:31:08.199 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate[75853]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-30T14:31:08.200 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate[75853]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T14:31:08.200 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate[75853]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-30T14:31:08.200 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:08 smithi031 podman[75840]: 2024-03-30 14:31:08.167936387 +0000 UTC m=+1.265541100 container died ab0117566734c09e47b299c7da2709dc7ebf0bf3cfe7ff861685063a5c0e1faf (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-30T14:31:08.460 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:08 smithi031 ceph-mon[67456]: pgmap v103: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 340 op/s; 182/1674 objects degraded (10.872%) 2024-03-30T14:31:08.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:08 smithi193 ceph-mon[52977]: pgmap v103: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 340 op/s; 182/1674 objects degraded (10.872%) 2024-03-30T14:31:09.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 205361 10.89 MB/sec execute 354 sec latency 37.587 ms 2024-03-30T14:31:09.738 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:09 smithi031 podman[75989]: 2024-03-30 14:31:09.558931941 +0000 UTC m=+1.384267141 container remove ab0117566734c09e47b299c7da2709dc7ebf0bf3cfe7ff861685063a5c0e1faf (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-activate, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-30T14:31:09.998 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:09 smithi031 podman[76048]: 2024-03-30 14:31:09.700362046 +0000 UTC m=+0.019216502 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:31:09.998 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:09 smithi031 podman[76048]: 2024-03-30 14:31:09.929667249 +0000 UTC m=+0.248521689 container create 507a07329c9876699ffdf07436f535ad924630e905147d5213f989e374403ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-30T14:31:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 205596 10.88 MB/sec execute 355 sec latency 55.835 ms 2024-03-30T14:31:10.509 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:10 smithi031 ceph-mon[67456]: pgmap v104: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 691 op/s; 184/1713 objects degraded (10.741%) 2024-03-30T14:31:10.510 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:10 smithi031 podman[76048]: 2024-03-30 14:31:10.313654697 +0000 UTC m=+0.632509157 container init 507a07329c9876699ffdf07436f535ad924630e905147d5213f989e374403ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-30T14:31:10.510 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:10 smithi031 podman[76048]: 2024-03-30 14:31:10.319631513 +0000 UTC m=+0.638485961 container start 507a07329c9876699ffdf07436f535ad924630e905147d5213f989e374403ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T14:31:10.510 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:10 smithi031 ceph-osd[76067]: -- 172.21.15.31:0/1131166652 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e353efd680 con 0x55e353ec6c00 2024-03-30T14:31:10.510 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:10 smithi031 bash[76048]: 507a07329c9876699ffdf07436f535ad924630e905147d5213f989e374403ee5 2024-03-30T14:31:10.510 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:10 smithi031 systemd[1]: Started Ceph osd.1 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:31:10.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:10 smithi193 ceph-mon[52977]: pgmap v104: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 691 op/s; 184/1713 objects degraded (10.741%) 2024-03-30T14:31:11.011 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:10 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[76063]: 2024-03-30T14:31:10.904+0000 7fb08b5937c0 -1 Falling back to public interface 2024-03-30T14:31:11.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 205849 10.86 MB/sec execute 356 sec latency 88.196 ms 2024-03-30T14:31:11.769 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:11 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:11.769 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:11 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:11.769 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:11 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:11.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:11 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:11.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:11 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:11.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:11 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 206121 10.84 MB/sec execute 357 sec latency 45.047 ms 2024-03-30T14:31:12.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:12 smithi031 ceph-mon[67456]: pgmap v105: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 571 op/s; 184/1713 objects degraded (10.741%) 2024-03-30T14:31:12.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:12 smithi193 ceph-mon[52977]: pgmap v105: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 571 op/s; 184/1713 objects degraded (10.741%) 2024-03-30T14:31:13.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 206393 10.82 MB/sec execute 358 sec latency 52.601 ms 2024-03-30T14:31:13.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:13.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:14.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:14.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:14.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 206746 10.83 MB/sec execute 359 sec latency 37.102 ms 2024-03-30T14:31:14.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:14 smithi031 ceph-mon[67456]: pgmap v106: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 513 op/s; 185/1686 objects degraded (10.973%) 2024-03-30T14:31:14.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:14 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 185/1686 objects degraded (10.973%), 24 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:15.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:14 smithi193 ceph-mon[52977]: pgmap v106: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 513 op/s; 185/1686 objects degraded (10.973%) 2024-03-30T14:31:15.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:14 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 185/1686 objects degraded (10.973%), 24 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 207140 10.83 MB/sec execute 360 sec latency 36.727 ms 2024-03-30T14:31:16.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 207492 10.83 MB/sec execute 361 sec latency 92.521 ms 2024-03-30T14:31:16.620 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:16 smithi031 ceph-mon[67456]: pgmap v107: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.9 MiB/s wr, 617 op/s; 195/1689 objects degraded (11.545%) 2024-03-30T14:31:17.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:16 smithi193 ceph-mon[52977]: pgmap v107: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.9 MiB/s wr, 617 op/s; 195/1689 objects degraded (11.545%) 2024-03-30T14:31:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 207845 10.82 MB/sec execute 362 sec latency 46.438 ms 2024-03-30T14:31:17.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:17 smithi031 ceph-mon[67456]: pgmap v108: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 440 op/s; 195/1710 objects degraded (11.404%) 2024-03-30T14:31:17.890 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:17 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[76063]: 2024-03-30T14:31:17.642+0000 7fb08b5937c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-30T14:31:18.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:17 smithi193 ceph-mon[52977]: pgmap v108: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 440 op/s; 195/1710 objects degraded (11.404%) 2024-03-30T14:31:18.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 208107 10.80 MB/sec execute 363 sec latency 37.447 ms 2024-03-30T14:31:19.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 208406 10.80 MB/sec execute 364 sec latency 50.038 ms 2024-03-30T14:31:19.178 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[76063]: 2024-03-30T14:31:19.046+0000 7fb08b5937c0 -1 osd.1 86 log_to_monitors true 2024-03-30T14:31:19.442 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:19.443 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:19.443 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:19 smithi031 ceph-mon[67456]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T14:31:19.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:19.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:19.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:19 smithi193 ceph-mon[52977]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T14:31:19.956 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:31:19 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[76063]: 2024-03-30T14:31:19.943+0000 7fb0789e3700 -1 osd.1 86 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:31:20.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 208672 10.79 MB/sec execute 365 sec latency 292.614 ms 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: pgmap v109: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.9 MiB/s wr, 616 op/s; 180/1641 objects degraded (10.969%) 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 180/1641 objects degraded (10.969%), 24 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: osdmap e89: 8 total, 7 up, 8 in 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.470 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:20.471 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:20.471 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:20 smithi031 ceph-mon[67456]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' 2024-03-30T14:31:20.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: pgmap v109: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.9 MiB/s wr, 616 op/s; 180/1641 objects degraded (10.969%) 2024-03-30T14:31:20.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 180/1641 objects degraded (10.969%), 24 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:20.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T14:31:20.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: osdmap e89: 8 total, 7 up, 8 in 2024-03-30T14:31:20.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:31:20.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: failed to encode map e89 with expected crc 2024-03-30T14:31:20.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:20.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:20.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:20 smithi193 ceph-mon[52977]: from='osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023]' entity='osd.1' 2024-03-30T14:31:21.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 208966 10.79 MB/sec execute 366 sec latency 243.267 ms 2024-03-30T14:31:21.599 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:21 smithi031 ceph-mon[67456]: failed to encode map e89 with expected crc 2024-03-30T14:31:21.599 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:21 smithi031 ceph-mon[67456]: failed to encode map e89 with expected crc 2024-03-30T14:31:21.599 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:21 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:31:21.599 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:21 smithi031 ceph-mon[67456]: failed to encode map e90 with expected crc 2024-03-30T14:31:21.599 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:21 smithi031 ceph-mon[67456]: osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023] boot 2024-03-30T14:31:21.600 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:21 smithi031 ceph-mon[67456]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T14:31:21.600 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:31:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:21 smithi193 ceph-mon[52977]: failed to encode map e89 with expected crc 2024-03-30T14:31:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:21 smithi193 ceph-mon[52977]: failed to encode map e89 with expected crc 2024-03-30T14:31:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:21 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:31:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:21 smithi193 ceph-mon[52977]: failed to encode map e90 with expected crc 2024-03-30T14:31:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:21 smithi193 ceph-mon[52977]: osd.1 [v2:172.21.15.31:6810/2066443023,v1:172.21.15.31:6811/2066443023] boot 2024-03-30T14:31:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:21 smithi193 ceph-mon[52977]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T14:31:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:31:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: 5 208966 10.76 MB/sec execute 367 sec latency 1243.310 ms 2024-03-30T14:31:22.747 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:22 smithi031 ceph-mon[67456]: pgmap v112: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.0 MiB/s wr, 552 op/s; 180/1641 objects degraded (10.969%) 2024-03-30T14:31:22.748 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:22 smithi031 ceph-mon[67456]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T14:31:22.748 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:22 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:22.748 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:22 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:22.748 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:22 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:22 smithi193 ceph-mon[52977]: pgmap v112: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.0 MiB/s wr, 552 op/s; 180/1641 objects degraded (10.969%) 2024-03-30T14:31:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:22 smithi193 ceph-mon[52977]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T14:31:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:22 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:22 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:22.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:22 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 209013 10.73 MB/sec execute 368 sec latency 2150.325 ms 2024-03-30T14:31:23.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.567 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T14:31:23.567 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.567 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.567 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.567 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:23 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e91 with expected crc 2024-03-30T14:31:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T14:31:23.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:23.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:23 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 209233 10.73 MB/sec execute 369 sec latency 223.397 ms 2024-03-30T14:31:24.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:24 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:24 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:24 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:24 smithi031 ceph-mon[67456]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:24 smithi031 ceph-mon[67456]: pgmap v115: 97 pgs: 2 peering, 23 active+undersized+degraded, 14 active+undersized, 58 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 763 KiB/s wr, 34 op/s; 163/1635 objects degraded (9.969%) 2024-03-30T14:31:24.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:24 smithi031 ceph-mon[67456]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T14:31:24.505 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:24 smithi031 ceph-mon[67456]: failed to encode map e93 with expected crc 2024-03-30T14:31:24.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:24 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:24 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:24 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:24 smithi193 ceph-mon[52977]: failed to encode map e92 with expected crc 2024-03-30T14:31:24.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:24 smithi193 ceph-mon[52977]: pgmap v115: 97 pgs: 2 peering, 23 active+undersized+degraded, 14 active+undersized, 58 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 763 KiB/s wr, 34 op/s; 163/1635 objects degraded (9.969%) 2024-03-30T14:31:24.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:24 smithi193 ceph-mon[52977]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T14:31:24.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:24 smithi193 ceph-mon[52977]: failed to encode map e93 with expected crc 2024-03-30T14:31:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 209464 10.71 MB/sec execute 370 sec latency 592.745 ms 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: failed to encode map e93 with expected crc 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 163/1635 objects degraded (9.969%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.530 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:25.531 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:25.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: failed to encode map e93 with expected crc 2024-03-30T14:31:25.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 163/1635 objects degraded (9.969%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: failed to encode map e94 with expected crc 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:25.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:25.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:25.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:25.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:25.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 209786 10.69 MB/sec execute 371 sec latency 608.920 ms 2024-03-30T14:31:26.545 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: failed to encode map e94 with expected crc 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: pgmap v118: 97 pgs: 85 active+clean, 3 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.7 MiB/s wr, 403 op/s; 141/1566 objects degraded (9.004%); 572 KiB/s, 73 keys/s, 19 objects/s recovering 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: pgmap v119: 97 pgs: 85 active+clean, 3 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.7 MiB/s wr, 438 op/s; 141/1566 objects degraded (9.004%); 690 KiB/s, 88 keys/s, 23 objects/s recovering 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.546 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:26 smithi031 ceph-mon[67456]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: failed to encode map e94 with expected crc 2024-03-30T14:31:26.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: pgmap v118: 97 pgs: 85 active+clean, 3 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.7 MiB/s wr, 403 op/s; 141/1566 objects degraded (9.004%); 572 KiB/s, 73 keys/s, 19 objects/s recovering 2024-03-30T14:31:26.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: pgmap v119: 97 pgs: 85 active+clean, 3 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.7 MiB/s wr, 438 op/s; 141/1566 objects degraded (9.004%); 690 KiB/s, 88 keys/s, 23 objects/s recovering 2024-03-30T14:31:26.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:26.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-03-30T14:31:26.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T14:31:26.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: failed to encode map e95 with expected crc 2024-03-30T14:31:26.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:26 smithi193 ceph-mon[52977]: failed to encode map e95 with expected crc 2024-03-30T14:31:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 210060 10.68 MB/sec execute 372 sec latency 374.701 ms 2024-03-30T14:31:27.555 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:27 smithi031 ceph-mon[67456]: failed to encode map e95 with expected crc 2024-03-30T14:31:27.556 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:27 smithi031 ceph-mon[67456]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.556 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:27 smithi031 ceph-mon[67456]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T14:31:27.556 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:27 smithi031 ceph-mon[67456]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.556 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:27 smithi031 ceph-mon[67456]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.556 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:27 smithi031 ceph-mon[67456]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.556 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:27 smithi031 ceph-mon[67456]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.582 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:31:27.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:27 smithi193 ceph-mon[52977]: failed to encode map e95 with expected crc 2024-03-30T14:31:27.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:27 smithi193 ceph-mon[52977]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:27 smithi193 ceph-mon[52977]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T14:31:27.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:27 smithi193 ceph-mon[52977]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:27 smithi193 ceph-mon[52977]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:27 smithi193 ceph-mon[52977]: failed to encode map e96 with expected crc 2024-03-30T14:31:27.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:27 smithi193 ceph-mon[52977]: failed to encode map e96 with expected crc 2024-03-30T14:31:28.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 210304 10.67 MB/sec execute 373 sec latency 785.114 ms 2024-03-30T14:31:28.509 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (5m) 10s ago 14m 23.9M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (105s) 10s ago 14m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (99s) 95s ago 14m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 10s ago 14m - - 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (8m) 10s ago 8m 40.5M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (8m) 95s ago 8m 14.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (4m) 10s ago 15m 504M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (3m) 95s ago 14m 449M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (3m) 10s ago 16m 48.6M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (2m) 95s ago 13m 36.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (8m) 10s ago 8m 289M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:31:28.510 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (8m) 95s ago 8m 82.5M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (5m) 10s ago 14m 20.0M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (4m) 95s ago 13m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (67s) 10s ago 13m 256M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (18s) 10s ago 12m 11.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (12m) 10s ago 12m 571M 1937M 17.2.0 e1d6a67b021e ebfb171a84df 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (11m) 10s ago 11m 375M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (11m) 95s ago 11m 546M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (11m) 95s ago 11m 402M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (10m) 95s ago 10m 357M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (10m) 95s ago 10m 453M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:31:28.511 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (3m) 10s ago 13m 63.2M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:31:28.587 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: failed to encode map e96 with expected crc 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: pgmap v122: 97 pgs: 86 active+clean, 2 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s wr, 51 op/s; 141/1581 objects degraded (8.918%); 0 B/s, 89 keys/s, 15 objects/s recovering 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:28.588 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:28.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: failed to encode map e96 with expected crc 2024-03-30T14:31:28.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: pgmap v122: 97 pgs: 86 active+clean, 2 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s wr, 51 op/s; 141/1581 objects degraded (8.918%); 0 B/s, 89 keys/s, 15 objects/s recovering 2024-03-30T14:31:28.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T14:31:28.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: failed to encode map e97 with expected crc 2024-03-30T14:31:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:29.083 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:31:29.083 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:31:29.083 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:31:29.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:31:29.084 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:31:29.085 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-30T14:31:29.085 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 6 2024-03-30T14:31:29.085 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:31:29.085 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:31:29.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 210598 10.67 MB/sec execute 374 sec latency 849.383 ms 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: failed to encode map e97 with expected crc 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: failed to encode map e97 with expected crc 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.592 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:29 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2592791240' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:29.603 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:31:29.603 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:31:29.603 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:31:29.603 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:31:29.603 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:31:29.603 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:31:29.603 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:31:29.604 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:31:29.604 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:31:29.604 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "8/23 daemons upgraded", 2024-03-30T14:31:29.604 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:31:29.604 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:31:29.604 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:31:29.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: failed to encode map e97 with expected crc 2024-03-30T14:31:29.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: failed to encode map e97 with expected crc 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: failed to encode map e98 with expected crc 2024-03-30T14:31:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:29 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2592791240' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:31:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: 5 211045 10.69 MB/sec execute 375 sec latency 55.755 ms 2024-03-30T14:31:30.213 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 29/1581 objects degraded (1.834%), 2 pgs degraded 2024-03-30T14:31:30.213 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:31:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:31:30.213 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 29/1581 objects degraded (1.834%), 2 pgs degraded 2024-03-30T14:31:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.0 is active+recovery_wait+undersized+degraded+remapped, acting [2,6] 2024-03-30T14:31:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.4 is active+recovering+undersized+degraded+remapped, acting [2,5] 2024-03-30T14:31:30.597 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:30 smithi031 ceph-mon[67456]: failed to encode map e98 with expected crc 2024-03-30T14:31:30.597 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:30 smithi031 ceph-mon[67456]: pgmap v125: 97 pgs: 92 active+clean, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.1 MiB/s wr, 463 op/s; 29/1581 objects degraded (1.834%); 232 KiB/s, 33 objects/s recovering 2024-03-30T14:31:30.597 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:30 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 141/1581 objects degraded (8.918%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:30.597 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:30 smithi031 ceph-mon[67456]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:30.598 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:30 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/222161312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:31:30.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:30 smithi193 ceph-mon[52977]: failed to encode map e98 with expected crc 2024-03-30T14:31:30.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:30 smithi193 ceph-mon[52977]: pgmap v125: 97 pgs: 92 active+clean, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.1 MiB/s wr, 463 op/s; 29/1581 objects degraded (1.834%); 232 KiB/s, 33 objects/s recovering 2024-03-30T14:31:30.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:30 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 141/1581 objects degraded (8.918%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:30.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:30 smithi193 ceph-mon[52977]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:31:30.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:30 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/222161312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:31:31.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 211519 10.67 MB/sec execute 376 sec latency 38.014 ms 2024-03-30T14:31:31.609 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:31 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/1581 objects degraded (1.834%), 2 pgs degraded) 2024-03-30T14:31:31.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:31 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/1581 objects degraded (1.834%), 2 pgs degraded) 2024-03-30T14:31:32.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 211858 10.66 MB/sec execute 377 sec latency 29.295 ms 2024-03-30T14:31:32.621 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:32 smithi031 ceph-mon[67456]: pgmap v126: 97 pgs: 95 active+clean, 2 peering; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.7 MiB/s wr, 373 op/s; 949 KiB/s, 33 objects/s recovering 2024-03-30T14:31:32.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:32 smithi193 ceph-mon[52977]: pgmap v126: 97 pgs: 95 active+clean, 2 peering; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.7 MiB/s wr, 373 op/s; 949 KiB/s, 33 objects/s recovering 2024-03-30T14:31:33.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 212399 10.69 MB/sec execute 378 sec latency 39.808 ms 2024-03-30T14:31:34.143 INFO:teuthology.orchestra.run.smithi031.stdout: 5 212929 10.71 MB/sec execute 379 sec latency 38.513 ms 2024-03-30T14:31:34.522 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:34 smithi031 ceph-mon[67456]: pgmap v127: 97 pgs: 96 active+clean, 1 peering; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.9 MiB/s wr, 349 op/s; 798 KiB/s, 28 objects/s recovering 2024-03-30T14:31:34.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:34 smithi193 ceph-mon[52977]: pgmap v127: 97 pgs: 96 active+clean, 1 peering; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.9 MiB/s wr, 349 op/s; 798 KiB/s, 28 objects/s recovering 2024-03-30T14:31:35.147 INFO:teuthology.orchestra.run.smithi031.stdout: 5 213266 10.69 MB/sec execute 380 sec latency 52.330 ms 2024-03-30T14:31:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 213652 10.68 MB/sec execute 381 sec latency 47.141 ms 2024-03-30T14:31:36.543 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:36 smithi031 ceph-mon[67456]: pgmap v128: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.1 MiB/s wr, 660 op/s; 512 KiB/s, 19 objects/s recovering 2024-03-30T14:31:36.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:36 smithi193 ceph-mon[52977]: pgmap v128: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.1 MiB/s wr, 660 op/s; 512 KiB/s, 19 objects/s recovering 2024-03-30T14:31:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 214028 10.69 MB/sec execute 382 sec latency 48.895 ms 2024-03-30T14:31:38.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 214323 10.69 MB/sec execute 383 sec latency 39.135 ms 2024-03-30T14:31:38.586 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:38 smithi031 ceph-mon[67456]: pgmap v129: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.0 MiB/s wr, 598 op/s; 441 KiB/s, 16 objects/s recovering 2024-03-30T14:31:38.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:38 smithi193 ceph-mon[52977]: pgmap v129: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.0 MiB/s wr, 598 op/s; 441 KiB/s, 16 objects/s recovering 2024-03-30T14:31:39.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 214715 10.69 MB/sec execute 384 sec latency 44.755 ms 2024-03-30T14:31:40.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 215174 10.67 MB/sec execute 385 sec latency 39.923 ms 2024-03-30T14:31:40.638 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:40 smithi031 ceph-mon[67456]: pgmap v130: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 398 KiB/s, 4 objects/s recovering 2024-03-30T14:31:40.639 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:40 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:40.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:40 smithi193 ceph-mon[52977]: pgmap v130: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 398 KiB/s, 4 objects/s recovering 2024-03-30T14:31:40.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:40 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:41.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 215550 10.67 MB/sec execute 386 sec latency 36.784 ms 2024-03-30T14:31:41.658 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:41 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:41.658 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:41 smithi031 ceph-mon[67456]: Upgrade: osd.2 is safe to restart 2024-03-30T14:31:41.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:41 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T14:31:41.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:41 smithi193 ceph-mon[52977]: Upgrade: osd.2 is safe to restart 2024-03-30T14:31:42.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 216106 10.69 MB/sec execute 387 sec latency 37.062 ms 2024-03-30T14:31:42.677 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:42 smithi031 ceph-mon[67456]: pgmap v131: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 504 op/s; 341 KiB/s, 3 objects/s recovering 2024-03-30T14:31:42.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:42 smithi193 ceph-mon[52977]: pgmap v131: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 504 op/s; 341 KiB/s, 3 objects/s recovering 2024-03-30T14:31:43.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 216603 10.71 MB/sec execute 388 sec latency 46.199 ms 2024-03-30T14:31:43.702 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:43.703 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:43.703 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:43.703 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:43.703 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:31:43.703 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:31:44.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:44.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:44.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:44.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:44.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:31:44.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:31:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 216927 10.69 MB/sec execute 389 sec latency 48.382 ms 2024-03-30T14:31:44.719 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:44 smithi031 ceph-mon[67456]: Upgrade: Updating osd.2 2024-03-30T14:31:44.719 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:44 smithi031 ceph-mon[67456]: Deploying daemon osd.2 on smithi031 2024-03-30T14:31:44.719 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:44 smithi031 ceph-mon[67456]: pgmap v132: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.3 MiB/s wr, 513 op/s 2024-03-30T14:31:45.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:44 smithi193 ceph-mon[52977]: Upgrade: Updating osd.2 2024-03-30T14:31:45.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:44 smithi193 ceph-mon[52977]: Deploying daemon osd.2 on smithi031 2024-03-30T14:31:45.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:44 smithi193 ceph-mon[52977]: pgmap v132: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.3 MiB/s wr, 513 op/s 2024-03-30T14:31:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 217322 10.69 MB/sec execute 390 sec latency 43.376 ms 2024-03-30T14:31:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 217649 10.69 MB/sec execute 391 sec latency 63.062 ms 2024-03-30T14:31:46.746 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:46 smithi031 ceph-mon[67456]: pgmap v133: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 737 op/s 2024-03-30T14:31:47.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:46 smithi193 ceph-mon[52977]: pgmap v133: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 737 op/s 2024-03-30T14:31:47.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 217974 10.69 MB/sec execute 392 sec latency 49.889 ms 2024-03-30T14:31:48.152 INFO:teuthology.orchestra.run.smithi031.stdout: 5 218329 10.69 MB/sec execute 393 sec latency 71.391 ms 2024-03-30T14:31:48.267 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:48 smithi031 systemd[1]: Stopping Ceph osd.2 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:31:48.769 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:48 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[42043]: 2024-03-30T14:31:48.610+0000 7fceee929700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:31:48.769 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:48 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[42043]: 2024-03-30T14:31:48.610+0000 7fceee929700 -1 osd.2 98 *** Got signal Terminated *** 2024-03-30T14:31:48.769 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:48 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[42043]: 2024-03-30T14:31:48.610+0000 7fceee929700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:31:48.769 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:48 smithi031 ceph-mon[67456]: pgmap v134: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.3 MiB/s wr, 485 op/s 2024-03-30T14:31:49.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:48 smithi193 ceph-mon[52977]: pgmap v134: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.3 MiB/s wr, 485 op/s 2024-03-30T14:31:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 218765 10.67 MB/sec execute 394 sec latency 47.726 ms 2024-03-30T14:31:49.779 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:49 smithi031 ceph-mon[67456]: osd.2 marked itself down and dead 2024-03-30T14:31:50.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:49 smithi193 ceph-mon[52977]: osd.2 marked itself down and dead 2024-03-30T14:31:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 219034 10.66 MB/sec execute 395 sec latency 544.569 ms 2024-03-30T14:31:50.546 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:50 smithi031 podman[78165]: 2024-03-30 14:31:50.280884211 +0000 UTC m=+1.845154091 container died ebfb171a84dfc2f51c5193c1fc6aa345515158b86a366e37ebd24c711677beb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., release=754, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vcs-type=git) 2024-03-30T14:31:50.798 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:50 smithi031 podman[78165]: 2024-03-30 14:31:50.750123898 +0000 UTC m=+2.314393784 container cleanup ebfb171a84dfc2f51c5193c1fc6aa345515158b86a366e37ebd24c711677beb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , ceph=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=) 2024-03-30T14:31:50.798 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:50 smithi031 bash[78165]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T14:31:50.798 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:50 smithi031 ceph-mon[67456]: pgmap v135: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 660 op/s 2024-03-30T14:31:50.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:50 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:31:50.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:50 smithi031 ceph-mon[67456]: osdmap e99: 8 total, 7 up, 8 in 2024-03-30T14:31:50.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:50 smithi031 ceph-mon[67456]: failed to encode map e99 with expected crc 2024-03-30T14:31:50.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:50 smithi031 ceph-mon[67456]: failed to encode map e99 with expected crc 2024-03-30T14:31:50.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:50 smithi031 ceph-mon[67456]: failed to encode map e99 with expected crc 2024-03-30T14:31:51.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:50 smithi193 ceph-mon[52977]: pgmap v135: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 660 op/s 2024-03-30T14:31:51.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:50 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:31:51.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:50 smithi193 ceph-mon[52977]: osdmap e99: 8 total, 7 up, 8 in 2024-03-30T14:31:51.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:50 smithi193 ceph-mon[52977]: failed to encode map e99 with expected crc 2024-03-30T14:31:51.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:50 smithi193 ceph-mon[52977]: failed to encode map e99 with expected crc 2024-03-30T14:31:51.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:50 smithi193 ceph-mon[52977]: failed to encode map e99 with expected crc 2024-03-30T14:31:51.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 219252 10.66 MB/sec execute 396 sec latency 1022.830 ms 2024-03-30T14:31:51.310 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:51 smithi031 podman[78179]: 2024-03-30 14:31:51.077126566 +0000 UTC m=+0.793716628 container remove ebfb171a84dfc2f51c5193c1fc6aa345515158b86a366e37ebd24c711677beb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., distribution-scope=public, build-date=2022-05-03T08:36:31.336870, release=754, GIT_BRANCH=HEAD, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-30T14:31:51.561 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:51 smithi031 podman[78268]: 2024-03-30 14:31:51.423409637 +0000 UTC m=+0.030639005 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:31:51.562 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:51 smithi031 podman[78268]: 2024-03-30 14:31:51.561790064 +0000 UTC m=+0.169019395 container create 90984fe29aeb01d213aaef6b0fd563bec8019e3e9eafeeecd1c81d4fa88b8a77 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: failed to encode map e100 with expected crc 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: osdmap e100: 8 total, 7 up, 8 in 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: failed to encode map e100 with expected crc 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: failed to encode map e100 with expected crc 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: failed to encode map e100 with expected crc 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: failed to encode map e100 with expected crc 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: failed to encode map e100 with expected crc 2024-03-30T14:31:51.816 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:51 smithi031 ceph-mon[67456]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.067 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:51 smithi031 podman[78268]: 2024-03-30 14:31:51.96648082 +0000 UTC m=+0.573710166 container init 90984fe29aeb01d213aaef6b0fd563bec8019e3e9eafeeecd1c81d4fa88b8a77 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T14:31:52.067 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:51 smithi031 podman[78268]: 2024-03-30 14:31:51.975628355 +0000 UTC m=+0.582857693 container start 90984fe29aeb01d213aaef6b0fd563bec8019e3e9eafeeecd1c81d4fa88b8a77 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-30T14:31:52.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: osdmap e100: 8 total, 7 up, 8 in 2024-03-30T14:31:52.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:51 smithi193 ceph-mon[52977]: failed to encode map e100 with expected crc 2024-03-30T14:31:52.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 219699 10.67 MB/sec execute 397 sec latency 35.886 ms 2024-03-30T14:31:52.317 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:52 smithi031 podman[78268]: 2024-03-30 14:31:52.1393198 +0000 UTC m=+0.746549135 container attach 90984fe29aeb01d213aaef6b0fd563bec8019e3e9eafeeecd1c81d4fa88b8a77 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-30T14:31:52.571 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:52 smithi031 podman[78268]: 2024-03-30 14:31:52.416344982 +0000 UTC m=+1.023574339 container died 90984fe29aeb01d213aaef6b0fd563bec8019e3e9eafeeecd1c81d4fa88b8a77 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326) 2024-03-30T14:31:52.824 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:52 smithi031 ceph-mon[67456]: pgmap v138: 97 pgs: 5 stale+active+clean, 92 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 697 op/s 2024-03-30T14:31:53.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:52 smithi193 ceph-mon[52977]: pgmap v138: 97 pgs: 5 stale+active+clean, 92 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 697 op/s 2024-03-30T14:31:53.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 220064 10.68 MB/sec execute 398 sec latency 57.036 ms 2024-03-30T14:31:53.836 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:53 smithi031 ceph-mon[67456]: pgmap v139: 97 pgs: 2 active+undersized, 1 active+undersized+degraded, 5 stale+active+clean, 89 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.8 MiB/s wr, 353 op/s; 12/1638 objects degraded (0.733%) 2024-03-30T14:31:53.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:53 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 12/1638 objects degraded (0.733%), 1 pg degraded (PG_DEGRADED) 2024-03-30T14:31:54.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:53 smithi193 ceph-mon[52977]: pgmap v139: 97 pgs: 2 active+undersized, 1 active+undersized+degraded, 5 stale+active+clean, 89 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.8 MiB/s wr, 353 op/s; 12/1638 objects degraded (0.733%) 2024-03-30T14:31:54.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:53 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 12/1638 objects degraded (0.733%), 1 pg degraded (PG_DEGRADED) 2024-03-30T14:31:54.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 220295 10.66 MB/sec execute 399 sec latency 60.745 ms 2024-03-30T14:31:54.358 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:54 smithi031 podman[78293]: 2024-03-30 14:31:54.194916544 +0000 UTC m=+1.771946103 container remove 90984fe29aeb01d213aaef6b0fd563bec8019e3e9eafeeecd1c81d4fa88b8a77 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-30T14:31:54.358 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:54 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.2.service: Deactivated successfully. 2024-03-30T14:31:54.358 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:54 smithi031 systemd[1]: Stopped Ceph osd.2 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:31:54.358 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:54 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.2.service: Consumed 3min 14.624s CPU time. 2024-03-30T14:31:54.609 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:54 smithi031 systemd[1]: Starting Ceph osd.2 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:31:54.866 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:54 smithi031 podman[78391]: 2024-03-30 14:31:54.68193146 +0000 UTC m=+0.019126319 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:31:54.867 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:54 smithi031 podman[78391]: 2024-03-30 14:31:54.803770423 +0000 UTC m=+0.140965267 container create 997156921ff9d1964251e673ec1700418b6ce35f99534f0bb80cd101bb599537 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T14:31:55.146 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 podman[78391]: 2024-03-30 14:31:55.107629558 +0000 UTC m=+0.444824420 container init 997156921ff9d1964251e673ec1700418b6ce35f99534f0bb80cd101bb599537 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-30T14:31:55.146 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 podman[78391]: 2024-03-30 14:31:55.112326837 +0000 UTC m=+0.449521683 container start 997156921ff9d1964251e673ec1700418b6ce35f99534f0bb80cd101bb599537 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-30T14:31:55.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 220541 10.64 MB/sec execute 400 sec latency 55.958 ms 2024-03-30T14:31:55.405 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 podman[78391]: 2024-03-30 14:31:55.240844397 +0000 UTC m=+0.578039249 container attach 997156921ff9d1964251e673ec1700418b6ce35f99534f0bb80cd101bb599537 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate[78405]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate[78405]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 bash[78391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 bash[78391]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 bash[78391]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 bash[78391]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 bash[78391]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 bash[78391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:31:55.658 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 bash[78391]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-30T14:31:55.659 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate[78405]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T14:31:55.659 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate[78405]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T14:31:55.659 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate[78405]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-30T14:31:55.659 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate[78405]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:31:55.659 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate[78405]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-30T14:31:55.911 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:55 smithi031 ceph-mon[67456]: pgmap v140: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.7 MiB/s wr, 614 op/s; 149/1659 objects degraded (8.981%) 2024-03-30T14:31:55.912 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:55 smithi031 podman[78391]: 2024-03-30 14:31:55.66545323 +0000 UTC m=+1.002648074 container died 997156921ff9d1964251e673ec1700418b6ce35f99534f0bb80cd101bb599537 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-30T14:31:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 220815 10.63 MB/sec execute 401 sec latency 58.286 ms 2024-03-30T14:31:56.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:55 smithi193 ceph-mon[52977]: pgmap v140: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.7 MiB/s wr, 614 op/s; 149/1659 objects degraded (8.981%) 2024-03-30T14:31:56.669 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:56 smithi031 podman[78542]: 2024-03-30 14:31:56.61397844 +0000 UTC m=+0.943591528 container remove 997156921ff9d1964251e673ec1700418b6ce35f99534f0bb80cd101bb599537 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-30T14:31:56.923 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:56 smithi031 podman[78596]: 2024-03-30 14:31:56.787639977 +0000 UTC m=+0.025677540 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:31:56.924 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:56 smithi031 podman[78596]: 2024-03-30 14:31:56.927285717 +0000 UTC m=+0.165323295 container create 2b3faa63c7084ddb0b213e3cce5d920816f22731bbdc796c885456fee51de1b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-30T14:31:57.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 221343 10.64 MB/sec execute 402 sec latency 52.917 ms 2024-03-30T14:31:57.430 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:57 smithi031 podman[78596]: 2024-03-30 14:31:57.299154639 +0000 UTC m=+0.537192193 container init 2b3faa63c7084ddb0b213e3cce5d920816f22731bbdc796c885456fee51de1b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-30T14:31:57.430 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:57 smithi031 podman[78596]: 2024-03-30 14:31:57.309845185 +0000 UTC m=+0.547882725 container start 2b3faa63c7084ddb0b213e3cce5d920816f22731bbdc796c885456fee51de1b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2) 2024-03-30T14:31:57.430 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:57 smithi031 ceph-osd[78614]: -- 172.21.15.31:0/2922650004 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563faccd2000 con 0x563fabee9000 2024-03-30T14:31:57.681 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:57 smithi031 bash[78596]: 2b3faa63c7084ddb0b213e3cce5d920816f22731bbdc796c885456fee51de1b7 2024-03-30T14:31:57.681 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:57 smithi031 systemd[1]: Started Ceph osd.2 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:31:57.932 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:31:57 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[78610]: 2024-03-30T14:31:57.908+0000 7f7ba402d7c0 -1 Falling back to public interface 2024-03-30T14:31:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 221798 10.66 MB/sec execute 403 sec latency 49.101 ms 2024-03-30T14:31:58.713 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:58 smithi031 ceph-mon[67456]: pgmap v141: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.1 MiB/s wr, 337 op/s; 149/1677 objects degraded (8.885%) 2024-03-30T14:31:58.713 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:58.713 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:58.713 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:58.713 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:58.713 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:58.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:58 smithi193 ceph-mon[52977]: pgmap v141: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.1 MiB/s wr, 337 op/s; 149/1677 objects degraded (8.885%) 2024-03-30T14:31:58.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:58.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:58.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:58.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:31:58.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:59.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 222335 10.65 MB/sec execute 404 sec latency 48.843 ms 2024-03-30T14:31:59.718 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:31:59 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 149/1677 objects degraded (8.885%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:31:59.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:31:59 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 149/1677 objects degraded (8.885%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:00.153 INFO:teuthology.orchestra.run.smithi031.stdout: 5 222942 10.67 MB/sec execute 405 sec latency 29.219 ms 2024-03-30T14:32:00.725 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:00 smithi031 ceph-mon[67456]: pgmap v142: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 421 op/s; 155/1671 objects degraded (9.276%) 2024-03-30T14:32:00.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:00 smithi193 ceph-mon[52977]: pgmap v142: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 421 op/s; 155/1671 objects degraded (9.276%) 2024-03-30T14:32:00.837 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:32:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: 5 223351 10.68 MB/sec execute 406 sec latency 50.313 ms 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (6m) 43s ago 15m 23.9M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (2m) 43s ago 15m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (2m) 2m ago 14m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 43s ago 14m - - 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (9m) 43s ago 9m 40.5M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (9m) 2m ago 9m 14.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:32:01.970 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (5m) 43s ago 16m 504M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (4m) 2m ago 14m 449M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (3m) 43s ago 16m 48.6M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (3m) 2m ago 14m 36.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (9m) 43s ago 8m 289M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (8m) 2m ago 8m 82.5M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (5m) 43s ago 14m 20.0M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (5m) 2m ago 14m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (100s) 43s ago 13m 256M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (52s) 43s ago 13m 11.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 starting - - - 1937M 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (12m) 43s ago 12m 375M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (12m) 2m ago 12m 546M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (11m) 2m ago 11m 402M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (11m) 2m ago 11m 357M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:32:01.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (10m) 2m ago 10m 453M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:32:01.972 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (4m) 43s ago 14m 63.2M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:32:02.154 INFO:teuthology.orchestra.run.smithi031.stdout: 5 223694 10.68 MB/sec execute 407 sec latency 75.769 ms 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:02.679 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:32:02.680 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:32:02.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:02.680 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:32:02.680 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-30T14:32:02.680 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 6 2024-03-30T14:32:02.680 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:32:02.680 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:32:02.761 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:02 smithi031 ceph-mon[67456]: pgmap v143: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.1 MiB/s wr, 508 op/s; 159/1674 objects degraded (9.498%) 2024-03-30T14:32:02.761 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:02 smithi031 ceph-mon[67456]: from='client.44213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:02.761 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:02 smithi031 ceph-mon[67456]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:02.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:02 smithi193 ceph-mon[52977]: pgmap v143: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.1 MiB/s wr, 508 op/s; 159/1674 objects degraded (9.498%) 2024-03-30T14:32:02.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:02 smithi193 ceph-mon[52977]: from='client.44213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:02.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:02 smithi193 ceph-mon[52977]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: 5 223946 10.66 MB/sec execute 408 sec latency 57.664 ms 2024-03-30T14:32:03.215 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "8/23 daemons upgraded", 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:32:03.216 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:32:03.543 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:03 smithi031 ceph-mon[67456]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:03.544 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:03 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/3357700882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:03.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:03 smithi193 ceph-mon[52977]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:03.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:03 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/3357700882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:03.923 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 159/1656 objects degraded (9.601%), 19 pgs degraded 2024-03-30T14:32:03.923 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:32:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:32:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T14:32:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: osd.2 (root=default,host=smithi031) is down 2024-03-30T14:32:03.924 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 159/1656 objects degraded (9.601%), 19 pgs degraded 2024-03-30T14:32:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-03-30T14:32:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-03-30T14:32:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-03-30T14:32:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-03-30T14:32:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-03-30T14:32:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-03-30T14:32:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-03-30T14:32:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-03-30T14:32:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-03-30T14:32:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-03-30T14:32:04.154 INFO:teuthology.orchestra.run.smithi031.stdout: 5 224225 10.65 MB/sec execute 409 sec latency 51.334 ms 2024-03-30T14:32:04.563 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:04 smithi031 ceph-mon[67456]: pgmap v144: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.8 MiB/s wr, 454 op/s; 159/1656 objects degraded (9.601%) 2024-03-30T14:32:04.563 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:04 smithi031 ceph-mon[67456]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:04.563 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:04 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2760204557' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:32:04.563 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:04 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 159/1656 objects degraded (9.601%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:04.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:04 smithi193 ceph-mon[52977]: pgmap v144: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.8 MiB/s wr, 454 op/s; 159/1656 objects degraded (9.601%) 2024-03-30T14:32:04.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:04 smithi193 ceph-mon[52977]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:04.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:04 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2760204557' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:32:04.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:04 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 159/1656 objects degraded (9.601%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:05.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 224430 10.63 MB/sec execute 410 sec latency 70.731 ms 2024-03-30T14:32:05.327 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:32:05 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[78610]: 2024-03-30T14:32:05.096+0000 7f7ba402d7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-30T14:32:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: 5 224674 10.63 MB/sec execute 411 sec latency 112.828 ms 2024-03-30T14:32:06.387 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:32:06 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[78610]: 2024-03-30T14:32:06.280+0000 7f7ba402d7c0 -1 osd.2 98 log_to_monitors true 2024-03-30T14:32:06.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:06 smithi031 ceph-mon[67456]: pgmap v145: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 687 op/s; 152/1635 objects degraded (9.297%) 2024-03-30T14:32:06.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:06.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:06.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:06 smithi031 ceph-mon[67456]: from='osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:32:06.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:06 smithi193 ceph-mon[52977]: pgmap v145: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 687 op/s; 152/1635 objects degraded (9.297%) 2024-03-30T14:32:06.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:06.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:06.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:06 smithi193 ceph-mon[52977]: from='osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:32:06.912 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:32:06 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[78610]: 2024-03-30T14:32:06.844+0000 7f7b95485700 -1 osd.2 98 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:32:07.156 INFO:teuthology.orchestra.run.smithi031.stdout: 5 224963 10.63 MB/sec execute 412 sec latency 51.011 ms 2024-03-30T14:32:08.063 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: from='osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:32:08.063 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: failed to encode map e101 with expected crc 2024-03-30T14:32:08.063 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: osdmap e101: 8 total, 7 up, 8 in 2024-03-30T14:32:08.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: from='osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:32:08.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: pgmap v147: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 598 op/s; 152/1626 objects degraded (9.348%) 2024-03-30T14:32:08.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: failed to encode map e101 with expected crc 2024-03-30T14:32:08.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: failed to encode map e101 with expected crc 2024-03-30T14:32:08.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:08.064 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:08.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: from='osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: failed to encode map e101 with expected crc 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: osdmap e101: 8 total, 7 up, 8 in 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: from='osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: pgmap v147: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 598 op/s; 152/1626 objects degraded (9.348%) 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: failed to encode map e101 with expected crc 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: failed to encode map e101 with expected crc 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:08.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 225147 10.62 MB/sec execute 413 sec latency 359.728 ms 2024-03-30T14:32:09.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:08 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:32:09.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:08 smithi193 ceph-mon[52977]: failed to encode map e102 with expected crc 2024-03-30T14:32:09.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:08 smithi193 ceph-mon[52977]: osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657] boot 2024-03-30T14:32:09.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:08 smithi193 ceph-mon[52977]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:32:09.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:08 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:32:09.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:08 smithi193 ceph-mon[52977]: failed to encode map e102 with expected crc 2024-03-30T14:32:09.110 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:08 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:32:09.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:08 smithi031 ceph-mon[67456]: failed to encode map e102 with expected crc 2024-03-30T14:32:09.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:08 smithi031 ceph-mon[67456]: osd.2 [v2:172.21.15.31:6818/3583976657,v1:172.21.15.31:6819/3583976657] boot 2024-03-30T14:32:09.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:08 smithi031 ceph-mon[67456]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:32:09.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:08 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:32:09.111 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:08 smithi031 ceph-mon[67456]: failed to encode map e102 with expected crc 2024-03-30T14:32:09.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 225147 10.59 MB/sec execute 414 sec latency 1359.770 ms 2024-03-30T14:32:10.020 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: pgmap v149: 97 pgs: 14 peering, 5 active+undersized, 7 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.9 MiB/s wr, 595 op/s; 57/1545 objects degraded (3.689%) 2024-03-30T14:32:10.020 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.020 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.021 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:09 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 152/1626 objects degraded (9.348%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:10.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: pgmap v149: 97 pgs: 14 peering, 5 active+undersized, 7 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.9 MiB/s wr, 595 op/s; 57/1545 objects degraded (3.689%) 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: failed to encode map e103 with expected crc 2024-03-30T14:32:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:09 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 152/1626 objects degraded (9.348%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:10.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 225195 10.57 MB/sec execute 415 sec latency 2228.718 ms 2024-03-30T14:32:11.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:10 smithi031 ceph-mon[67456]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:32:11.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:10 smithi031 ceph-mon[67456]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:10 smithi031 ceph-mon[67456]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:10 smithi031 ceph-mon[67456]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:10 smithi031 ceph-mon[67456]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.031 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:10 smithi031 ceph-mon[67456]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.032 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:10 smithi031 ceph-mon[67456]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:10 smithi193 ceph-mon[52977]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:32:11.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:10 smithi193 ceph-mon[52977]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:10 smithi193 ceph-mon[52977]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:10 smithi193 ceph-mon[52977]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:10 smithi193 ceph-mon[52977]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:10 smithi193 ceph-mon[52977]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:10 smithi193 ceph-mon[52977]: failed to encode map e104 with expected crc 2024-03-30T14:32:11.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 225517 10.56 MB/sec execute 416 sec latency 46.367 ms 2024-03-30T14:32:12.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:11 smithi031 ceph-mon[67456]: pgmap v152: 97 pgs: 2 remapped+peering, 14 peering, 3 active+undersized, 5 active+undersized+degraded, 73 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 352 op/s; 31/1542 objects degraded (2.010%) 2024-03-30T14:32:12.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:11 smithi031 ceph-mon[67456]: failed to encode map e105 with expected crc 2024-03-30T14:32:12.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:11 smithi031 ceph-mon[67456]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:32:12.043 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:11 smithi031 ceph-mon[67456]: failed to encode map e105 with expected crc 2024-03-30T14:32:12.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:11 smithi193 ceph-mon[52977]: pgmap v152: 97 pgs: 2 remapped+peering, 14 peering, 3 active+undersized, 5 active+undersized+degraded, 73 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 352 op/s; 31/1542 objects degraded (2.010%) 2024-03-30T14:32:12.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:11 smithi193 ceph-mon[52977]: failed to encode map e105 with expected crc 2024-03-30T14:32:12.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:11 smithi193 ceph-mon[52977]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:32:12.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:11 smithi193 ceph-mon[52977]: failed to encode map e105 with expected crc 2024-03-30T14:32:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 225841 10.54 MB/sec execute 417 sec latency 310.532 ms 2024-03-30T14:32:13.082 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.082 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:32:13.082 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.082 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.082 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.082 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: pgmap v155: 97 pgs: 2 remapped+peering, 14 peering, 81 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 55 op/s; 72 KiB/s, 6 objects/s recovering 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: pgmap v156: 97 pgs: 2 remapped+peering, 14 peering, 81 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 55 op/s; 72 KiB/s, 6 objects/s recovering 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:13.083 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-03-30T14:32:13.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.084 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:13.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 226063 10.52 MB/sec execute 418 sec latency 1000.407 ms 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: failed to encode map e106 with expected crc 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: pgmap v155: 97 pgs: 2 remapped+peering, 14 peering, 81 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 55 op/s; 72 KiB/s, 6 objects/s recovering 2024-03-30T14:32:13.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: pgmap v156: 97 pgs: 2 remapped+peering, 14 peering, 81 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 55 op/s; 72 KiB/s, 6 objects/s recovering 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:13.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:13.849 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: failed to encode map e107 with expected crc 2024-03-30T14:32:13.849 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: failed to encode map e107 with expected crc 2024-03-30T14:32:13.849 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:32:13.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: failed to encode map e107 with expected crc 2024-03-30T14:32:13.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: failed to encode map e107 with expected crc 2024-03-30T14:32:13.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: failed to encode map e107 with expected crc 2024-03-30T14:32:13.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: failed to encode map e107 with expected crc 2024-03-30T14:32:13.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:13 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/1542 objects degraded (2.010%), 5 pgs degraded) 2024-03-30T14:32:14.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: failed to encode map e107 with expected crc 2024-03-30T14:32:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: failed to encode map e107 with expected crc 2024-03-30T14:32:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:32:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: failed to encode map e107 with expected crc 2024-03-30T14:32:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: failed to encode map e107 with expected crc 2024-03-30T14:32:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: failed to encode map e107 with expected crc 2024-03-30T14:32:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: failed to encode map e107 with expected crc 2024-03-30T14:32:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:13 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/1542 objects degraded (2.010%), 5 pgs degraded) 2024-03-30T14:32:14.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 226349 10.53 MB/sec execute 419 sec latency 312.423 ms 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: failed to encode map e108 with expected crc 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: failed to encode map e108 with expected crc 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: failed to encode map e108 with expected crc 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: failed to encode map e108 with expected crc 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: failed to encode map e108 with expected crc 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: failed to encode map e108 with expected crc 2024-03-30T14:32:14.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:14 smithi031 ceph-mon[67456]: pgmap v159: 97 pgs: 1 active+undersized+remapped, 2 remapped+peering, 1 peering, 93 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 3.5 MiB/s wr, 422 op/s; 2.2 MiB/s, 40 objects/s recovering 2024-03-30T14:32:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 226646 10.53 MB/sec execute 420 sec latency 1022.958 ms 2024-03-30T14:32:15.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: failed to encode map e108 with expected crc 2024-03-30T14:32:15.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:32:15.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: failed to encode map e108 with expected crc 2024-03-30T14:32:15.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: failed to encode map e108 with expected crc 2024-03-30T14:32:15.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: failed to encode map e108 with expected crc 2024-03-30T14:32:15.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: failed to encode map e108 with expected crc 2024-03-30T14:32:15.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: failed to encode map e108 with expected crc 2024-03-30T14:32:15.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:14 smithi193 ceph-mon[52977]: pgmap v159: 97 pgs: 1 active+undersized+remapped, 2 remapped+peering, 1 peering, 93 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 3.5 MiB/s wr, 422 op/s; 2.2 MiB/s, 40 objects/s recovering 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: failed to encode map e109 with expected crc 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: failed to encode map e109 with expected crc 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: failed to encode map e109 with expected crc 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: failed to encode map e109 with expected crc 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: failed to encode map e109 with expected crc 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: failed to encode map e109 with expected crc 2024-03-30T14:32:15.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:15 smithi031 ceph-mon[67456]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 226965 10.53 MB/sec execute 421 sec latency 58.786 ms 2024-03-30T14:32:16.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:32:16.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:15 smithi193 ceph-mon[52977]: failed to encode map e109 with expected crc 2024-03-30T14:32:16.899 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:16 smithi031 ceph-mon[67456]: pgmap v161: 97 pgs: 1 active+undersized+remapped, 2 peering, 94 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 354 op/s; 2.1 MiB/s, 40 objects/s recovering 2024-03-30T14:32:17.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 227287 10.52 MB/sec execute 422 sec latency 55.770 ms 2024-03-30T14:32:17.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:16 smithi193 ceph-mon[52977]: pgmap v161: 97 pgs: 1 active+undersized+remapped, 2 peering, 94 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 354 op/s; 2.1 MiB/s, 40 objects/s recovering 2024-03-30T14:32:18.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 227521 10.51 MB/sec execute 423 sec latency 38.122 ms 2024-03-30T14:32:19.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 227856 10.51 MB/sec execute 424 sec latency 41.326 ms 2024-03-30T14:32:19.436 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:19 smithi031 ceph-mon[67456]: pgmap v162: 97 pgs: 1 active+undersized+remapped, 2 peering, 94 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.2 MiB/s wr, 378 op/s; 1.4 MiB/s, 26 objects/s recovering 2024-03-30T14:32:19.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:19 smithi193 ceph-mon[52977]: pgmap v162: 97 pgs: 1 active+undersized+remapped, 2 peering, 94 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.2 MiB/s wr, 378 op/s; 1.4 MiB/s, 26 objects/s recovering 2024-03-30T14:32:20.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 228234 10.51 MB/sec execute 425 sec latency 72.554 ms 2024-03-30T14:32:21.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 228528 10.50 MB/sec execute 426 sec latency 46.171 ms 2024-03-30T14:32:21.463 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:21 smithi031 ceph-mon[67456]: pgmap v163: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.3 MiB/s wr, 455 op/s; 1.1 MiB/s, 21 objects/s recovering 2024-03-30T14:32:21.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:21 smithi193 ceph-mon[52977]: pgmap v163: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.3 MiB/s wr, 455 op/s; 1.1 MiB/s, 21 objects/s recovering 2024-03-30T14:32:22.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 228825 10.50 MB/sec execute 427 sec latency 60.216 ms 2024-03-30T14:32:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 229233 10.49 MB/sec execute 428 sec latency 42.490 ms 2024-03-30T14:32:23.484 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:23 smithi031 ceph-mon[67456]: pgmap v164: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.5 MiB/s wr, 315 op/s; 115 KiB/s, 5 objects/s recovering 2024-03-30T14:32:23.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:23 smithi193 ceph-mon[52977]: pgmap v164: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.5 MiB/s wr, 315 op/s; 115 KiB/s, 5 objects/s recovering 2024-03-30T14:32:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 229559 10.47 MB/sec execute 429 sec latency 49.803 ms 2024-03-30T14:32:25.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 229873 10.48 MB/sec execute 430 sec latency 100.373 ms 2024-03-30T14:32:25.538 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:25 smithi031 ceph-mon[67456]: pgmap v165: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.0 MiB/s wr, 327 op/s; 98 KiB/s, 4 objects/s recovering 2024-03-30T14:32:25.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:25 smithi193 ceph-mon[52977]: pgmap v165: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.0 MiB/s wr, 327 op/s; 98 KiB/s, 4 objects/s recovering 2024-03-30T14:32:26.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 230239 10.49 MB/sec execute 431 sec latency 47.856 ms 2024-03-30T14:32:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 230643 10.49 MB/sec execute 432 sec latency 50.175 ms 2024-03-30T14:32:27.594 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:27 smithi031 ceph-mon[67456]: pgmap v166: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 436 op/s 2024-03-30T14:32:27.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:27 smithi193 ceph-mon[52977]: pgmap v166: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 436 op/s 2024-03-30T14:32:28.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 230938 10.49 MB/sec execute 433 sec latency 37.486 ms 2024-03-30T14:32:28.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:28.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:28 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:28.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:28 smithi031 ceph-mon[67456]: Upgrade: osd.3 is safe to restart 2024-03-30T14:32:28.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:28.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:28.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:28.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:28 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T14:32:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:28 smithi193 ceph-mon[52977]: Upgrade: osd.3 is safe to restart 2024-03-30T14:32:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:29.158 INFO:teuthology.orchestra.run.smithi031.stdout: 5 231163 10.48 MB/sec execute 434 sec latency 39.196 ms 2024-03-30T14:32:29.625 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:29 smithi031 ceph-mon[67456]: pgmap v167: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.4 MiB/s wr, 436 op/s 2024-03-30T14:32:29.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:29 smithi193 ceph-mon[52977]: pgmap v167: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.4 MiB/s wr, 436 op/s 2024-03-30T14:32:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 231457 10.47 MB/sec execute 435 sec latency 48.160 ms 2024-03-30T14:32:31.139 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:31 smithi031 ceph-mon[67456]: Upgrade: Updating osd.3 2024-03-30T14:32:31.139 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:31.139 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:32:31.139 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:32:31.139 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:31 smithi031 ceph-mon[67456]: Deploying daemon osd.3 on smithi031 2024-03-30T14:32:31.139 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:31 smithi031 ceph-mon[67456]: pgmap v168: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-03-30T14:32:31.157 INFO:teuthology.orchestra.run.smithi031.stdout: 5 231786 10.46 MB/sec execute 436 sec latency 43.296 ms 2024-03-30T14:32:31.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:31 smithi193 ceph-mon[52977]: Upgrade: Updating osd.3 2024-03-30T14:32:31.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:31.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:32:31.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:32:31.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:31 smithi193 ceph-mon[52977]: Deploying daemon osd.3 on smithi031 2024-03-30T14:32:31.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:31 smithi193 ceph-mon[52977]: pgmap v168: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-03-30T14:32:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: 5 232141 10.47 MB/sec execute 437 sec latency 48.568 ms 2024-03-30T14:32:33.158 INFO:teuthology.orchestra.run.smithi031.stdout: 5 232628 10.46 MB/sec execute 438 sec latency 46.660 ms 2024-03-30T14:32:33.415 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:33 smithi031 ceph-mon[67456]: pgmap v169: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.6 MiB/s wr, 440 op/s 2024-03-30T14:32:33.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:33 smithi193 ceph-mon[52977]: pgmap v169: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.6 MiB/s wr, 440 op/s 2024-03-30T14:32:34.158 INFO:teuthology.orchestra.run.smithi031.stdout: 5 233057 10.45 MB/sec execute 439 sec latency 56.469 ms 2024-03-30T14:32:34.478 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:32:35.158 INFO:teuthology.orchestra.run.smithi031.stdout: 5 233363 10.45 MB/sec execute 440 sec latency 52.775 ms 2024-03-30T14:32:35.444 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:35 smithi031 ceph-mon[67456]: pgmap v170: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.5 MiB/s wr, 585 op/s 2024-03-30T14:32:35.444 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:35 smithi031 ceph-mon[67456]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:35.622 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:32:35.622 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (6m) 29s ago 15m 23.9M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:32:35.622 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (2m) 29s ago 15m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:32:35.622 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (2m) 2m ago 15m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:32:35.622 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 29s ago 15m - - 2024-03-30T14:32:35.622 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (9m) 29s ago 9m 40.5M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:32:35.622 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (9m) 2m ago 9m 14.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (5m) 29s ago 17m 506M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (4m) 2m ago 15m 449M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (4m) 29s ago 17m 50.3M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (3m) 2m ago 15m 36.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (9m) 29s ago 9m 285M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (9m) 2m ago 9m 82.5M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (6m) 29s ago 15m 17.2M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (5m) 2m ago 14m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (2m) 29s ago 14m 313M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (85s) 29s ago 13m 243M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (38s) 29s ago 13m 11.8M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (12m) 29s ago 12m 413M 1937M 17.2.0 e1d6a67b021e bcbbc2919133 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (12m) 2m ago 12m 546M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:32:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (12m) 2m ago 12m 402M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:32:35.624 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (11m) 2m ago 11m 357M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:32:35.624 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (11m) 2m ago 11m 453M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:32:35.624 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (4m) 29s ago 14m 69.9M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:32:35.701 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:35 smithi031 systemd[1]: Stopping Ceph osd.3 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:32:35.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:35 smithi193 ceph-mon[52977]: pgmap v170: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.5 MiB/s wr, 585 op/s 2024-03-30T14:32:35.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:35 smithi193 ceph-mon[52977]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 233678 10.46 MB/sec execute 441 sec latency 70.014 ms 2024-03-30T14:32:36.203 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:35 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[45910]: 2024-03-30T14:32:35.979+0000 7fc941b59700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:32:36.203 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:35 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[45910]: 2024-03-30T14:32:35.979+0000 7fc941b59700 -1 osd.3 109 *** Got signal Terminated *** 2024-03-30T14:32:36.203 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:35 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[45910]: 2024-03-30T14:32:35.979+0000 7fc941b59700 -1 osd.3 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:32:36.265 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:32:36.265 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 3 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:32:36.266 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:32:36.267 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-30T14:32:36.267 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 7 2024-03-30T14:32:36.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:32:36.267 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:32:36.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:36 smithi031 ceph-mon[67456]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:36.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:36 smithi031 ceph-mon[67456]: from='client.44241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:36.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:36 smithi031 ceph-mon[67456]: osd.3 marked itself down and dead 2024-03-30T14:32:36.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:36 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/3699995810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:36.669 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:32:36.669 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:32:36.669 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "9/23 daemons upgraded", 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:32:36.670 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:32:36.735 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:36 smithi193 ceph-mon[52977]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:36.735 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:36 smithi193 ceph-mon[52977]: from='client.44241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:36.735 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:36 smithi193 ceph-mon[52977]: osd.3 marked itself down and dead 2024-03-30T14:32:36.735 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:36 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/3699995810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:32:37.078 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down 2024-03-30T14:32:37.078 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:32:37.078 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:32:37.078 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T14:32:37.079 INFO:teuthology.orchestra.run.smithi031.stdout: osd.3 (root=default,host=smithi031) is down 2024-03-30T14:32:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 233787 10.44 MB/sec execute 442 sec latency 776.824 ms 2024-03-30T14:32:37.361 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:37 smithi031 podman[80761]: 2024-03-30 14:32:37.059446059 +0000 UTC m=+1.290582364 container died bcbbc29191335e417fd9929d8c99ea91b062fa98f23815b18e2b284cdc45cff5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_CLEAN=True, vcs-type=git, ceph=True, com.redhat.component=centos-stream-container, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0) 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: pgmap v171: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 547 op/s 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: osdmap e110: 8 total, 7 up, 8 in 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:37 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/4227119834' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: pgmap v171: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 547 op/s 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: osdmap e110: 8 total, 7 up, 8 in 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: failed to encode map e110 with expected crc 2024-03-30T14:32:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:37 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/4227119834' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:32:37.907 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:37 smithi031 podman[80761]: 2024-03-30 14:32:37.88534307 +0000 UTC m=+2.116479375 container cleanup bcbbc29191335e417fd9929d8c99ea91b062fa98f23815b18e2b284cdc45cff5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, ceph=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , distribution-scope=public, name=centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-30T14:32:37.907 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:37 smithi031 bash[80761]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T14:32:38.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 234248 10.44 MB/sec execute 443 sec latency 1045.972 ms 2024-03-30T14:32:38.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:38.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: osdmap e111: 8 total, 7 up, 8 in 2024-03-30T14:32:38.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.409 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.410 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.662 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:38 smithi031 podman[80842]: 2024-03-30 14:32:38.503344858 +0000 UTC m=+1.440007163 container remove bcbbc29191335e417fd9929d8c99ea91b062fa98f23815b18e2b284cdc45cff5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.openshift.expose-services=, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-30T14:32:38.662 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.662 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.662 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:38 smithi031 ceph-mon[67456]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: osdmap e111: 8 total, 7 up, 8 in 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:38 smithi193 ceph-mon[52977]: failed to encode map e111 with expected crc 2024-03-30T14:32:39.165 INFO:teuthology.orchestra.run.smithi031.stdout: 5 234566 10.44 MB/sec execute 444 sec latency 31.432 ms 2024-03-30T14:32:39.166 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:39 smithi031 podman[80927]: 2024-03-30 14:32:39.042477358 +0000 UTC m=+0.027952744 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:32:39.417 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:39 smithi031 podman[80927]: 2024-03-30 14:32:39.171405198 +0000 UTC m=+0.156880574 container create 0a7519998db75be4ccb3d1b3fa55e090910b683cbc8549ea73df25d868acb2da (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-30T14:32:39.417 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:39 smithi031 ceph-mon[67456]: pgmap v174: 97 pgs: 14 peering, 11 stale+active+clean, 72 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-03-30T14:32:39.417 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:39 smithi031 ceph-mon[67456]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:32:39.672 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:39 smithi031 podman[80927]: 2024-03-30 14:32:39.553291024 +0000 UTC m=+0.538766400 container init 0a7519998db75be4ccb3d1b3fa55e090910b683cbc8549ea73df25d868acb2da (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-30T14:32:39.672 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:39 smithi031 podman[80927]: 2024-03-30 14:32:39.560048255 +0000 UTC m=+0.545523630 container start 0a7519998db75be4ccb3d1b3fa55e090910b683cbc8549ea73df25d868acb2da (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-30T14:32:39.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:39 smithi193 ceph-mon[52977]: pgmap v174: 97 pgs: 14 peering, 11 stale+active+clean, 72 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-03-30T14:32:39.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:39 smithi193 ceph-mon[52977]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:32:39.923 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:39 smithi031 podman[80927]: 2024-03-30 14:32:39.718775171 +0000 UTC m=+0.704250544 container attach 0a7519998db75be4ccb3d1b3fa55e090910b683cbc8549ea73df25d868acb2da (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc) 2024-03-30T14:32:40.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 234921 10.45 MB/sec execute 445 sec latency 27.000 ms 2024-03-30T14:32:40.175 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:40 smithi031 podman[80927]: 2024-03-30 14:32:40.031109423 +0000 UTC m=+1.016584790 container died 0a7519998db75be4ccb3d1b3fa55e090910b683cbc8549ea73df25d868acb2da (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-30T14:32:40.677 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:40 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 129/1620 objects degraded (7.963%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:40.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:40 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 129/1620 objects degraded (7.963%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 235289 10.44 MB/sec execute 446 sec latency 46.427 ms 2024-03-30T14:32:41.506 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:41 smithi031 podman[80952]: 2024-03-30 14:32:41.495845542 +0000 UTC m=+1.447993424 container remove 0a7519998db75be4ccb3d1b3fa55e090910b683cbc8549ea73df25d868acb2da (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:32:41.506 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:41 smithi031 ceph-mon[67456]: pgmap v175: 97 pgs: 10 active+undersized, 17 active+undersized+degraded, 14 peering, 56 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 MiB/s wr, 584 op/s; 129/1620 objects degraded (7.963%) 2024-03-30T14:32:41.762 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:41 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.3.service: Deactivated successfully. 2024-03-30T14:32:41.762 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:41 smithi031 systemd[1]: Stopped Ceph osd.3 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:32:41.762 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:41 smithi031 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.3.service: Consumed 4min 32.814s CPU time. 2024-03-30T14:32:41.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:41 smithi193 ceph-mon[52977]: pgmap v175: 97 pgs: 10 active+undersized, 17 active+undersized+degraded, 14 peering, 56 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 MiB/s wr, 584 op/s; 129/1620 objects degraded (7.963%) 2024-03-30T14:32:42.018 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:41 smithi031 systemd[1]: Starting Ceph osd.3 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:32:42.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 235605 10.44 MB/sec execute 447 sec latency 39.654 ms 2024-03-30T14:32:42.272 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:42 smithi031 podman[81085]: 2024-03-30 14:32:42.008407171 +0000 UTC m=+0.022354360 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:32:42.272 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:42 smithi031 podman[81085]: 2024-03-30 14:32:42.240123192 +0000 UTC m=+0.254070364 container create d45b0d8302102a6e9dc01efc54bf36dbb314dcae0af9cbc9598616e1a7702934 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-30T14:32:42.523 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:42 smithi031 ceph-mon[67456]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:32:42.774 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:42 smithi031 podman[81085]: 2024-03-30 14:32:42.686444674 +0000 UTC m=+0.700391862 container init d45b0d8302102a6e9dc01efc54bf36dbb314dcae0af9cbc9598616e1a7702934 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-30T14:32:42.774 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:42 smithi031 podman[81085]: 2024-03-30 14:32:42.696665779 +0000 UTC m=+0.710612956 container start d45b0d8302102a6e9dc01efc54bf36dbb314dcae0af9cbc9598616e1a7702934 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:32:42.833 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:42 smithi193 ceph-mon[52977]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:32:43.035 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:42 smithi031 podman[81085]: 2024-03-30 14:32:42.887184542 +0000 UTC m=+0.901131723 container attach d45b0d8302102a6e9dc01efc54bf36dbb314dcae0af9cbc9598616e1a7702934 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate, RELEASE=squid-4e50bbc, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-30T14:32:43.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 235957 10.44 MB/sec execute 448 sec latency 40.410 ms 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate[81099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate[81099]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 bash[81085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 bash[81085]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 bash[81085]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 bash[81085]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 bash[81085]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 bash[81085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 bash[81085]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate[81099]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate[81099]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate[81099]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate[81099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T14:32:43.287 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate[81099]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-30T14:32:43.288 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:43 smithi031 podman[81085]: 2024-03-30 14:32:43.286914578 +0000 UTC m=+1.300861766 container died d45b0d8302102a6e9dc01efc54bf36dbb314dcae0af9cbc9598616e1a7702934 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-30T14:32:43.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-mon[67456]: pgmap v176: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 380 op/s; 214/1662 objects degraded (12.876%) 2024-03-30T14:32:43.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:43.802 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:44.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:43 smithi193 ceph-mon[52977]: pgmap v176: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 380 op/s; 214/1662 objects degraded (12.876%) 2024-03-30T14:32:44.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:44.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 236425 10.43 MB/sec execute 449 sec latency 52.613 ms 2024-03-30T14:32:44.829 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:44 smithi031 podman[81220]: 2024-03-30 14:32:44.615195412 +0000 UTC m=+1.314593623 container remove d45b0d8302102a6e9dc01efc54bf36dbb314dcae0af9cbc9598616e1a7702934 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-30T14:32:45.095 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:44 smithi031 podman[81272]: 2024-03-30 14:32:44.778818577 +0000 UTC m=+0.024027963 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:32:45.095 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:44 smithi031 podman[81272]: 2024-03-30 14:32:44.986111597 +0000 UTC m=+0.231320938 container create 1de380dc2f05fa1d92969055d2ccf4a8f2e18919087ce5132241fad41a246b82 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-03-30T14:32:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 236762 10.42 MB/sec execute 450 sec latency 41.975 ms 2024-03-30T14:32:45.601 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:45 smithi031 ceph-mon[67456]: pgmap v177: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.7 MiB/s wr, 528 op/s; 218/1716 objects degraded (12.704%) 2024-03-30T14:32:45.601 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:45 smithi031 podman[81272]: 2024-03-30 14:32:45.354244943 +0000 UTC m=+0.599454306 container init 1de380dc2f05fa1d92969055d2ccf4a8f2e18919087ce5132241fad41a246b82 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-30T14:32:45.601 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:45 smithi031 podman[81272]: 2024-03-30 14:32:45.36283424 +0000 UTC m=+0.608043593 container start 1de380dc2f05fa1d92969055d2ccf4a8f2e18919087ce5132241fad41a246b82 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-03-30T14:32:45.601 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:45 smithi031 ceph-osd[81291]: -- 172.21.15.31:0/389031284 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a62f1b5680 con 0x55a62f17f000 2024-03-30T14:32:45.602 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:45 smithi031 bash[81272]: 1de380dc2f05fa1d92969055d2ccf4a8f2e18919087ce5132241fad41a246b82 2024-03-30T14:32:45.602 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:45 smithi031 systemd[1]: Started Ceph osd.3 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:32:46.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:45 smithi193 ceph-mon[52977]: pgmap v177: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.7 MiB/s wr, 528 op/s; 218/1716 objects degraded (12.704%) 2024-03-30T14:32:46.119 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:45 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:32:45.952+0000 7f7234c9a7c0 -1 Falling back to public interface 2024-03-30T14:32:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: 5 237112 10.43 MB/sec execute 451 sec latency 51.824 ms 2024-03-30T14:32:46.622 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:46 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:46.622 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:46 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:46.622 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:46 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:32:47.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:46 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:47.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:46 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:47.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:46 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:32:47.164 INFO:teuthology.orchestra.run.smithi031.stdout: 5 237587 10.44 MB/sec execute 452 sec latency 41.008 ms 2024-03-30T14:32:47.639 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:47 smithi031 ceph-mon[67456]: pgmap v178: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 348 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 5.2 MiB/s wr, 517 op/s; 222/1713 objects degraded (12.960%) 2024-03-30T14:32:47.639 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:47 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 222/1713 objects degraded (12.960%), 27 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:48.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:47 smithi193 ceph-mon[52977]: pgmap v178: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 348 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 5.2 MiB/s wr, 517 op/s; 222/1713 objects degraded (12.960%) 2024-03-30T14:32:48.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:47 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 222/1713 objects degraded (12.960%), 27 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:48.169 INFO:teuthology.orchestra.run.smithi031.stdout: 5 237985 10.44 MB/sec execute 453 sec latency 40.649 ms 2024-03-30T14:32:49.170 INFO:teuthology.orchestra.run.smithi031.stdout: 5 238155 10.43 MB/sec execute 454 sec latency 72.337 ms 2024-03-30T14:32:49.655 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:49 smithi031 ceph-mon[67456]: pgmap v179: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 410 op/s; 207/1632 objects degraded (12.684%) 2024-03-30T14:32:50.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:49 smithi193 ceph-mon[52977]: pgmap v179: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 410 op/s; 207/1632 objects degraded (12.684%) 2024-03-30T14:32:50.169 INFO:teuthology.orchestra.run.smithi031.stdout: 5 238409 10.42 MB/sec execute 455 sec latency 44.081 ms 2024-03-30T14:32:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: 5 238624 10.41 MB/sec execute 456 sec latency 61.749 ms 2024-03-30T14:32:51.732 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:51 smithi031 ceph-mon[67456]: pgmap v180: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 212/1638 objects degraded (12.943%) 2024-03-30T14:32:52.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:51 smithi193 ceph-mon[52977]: pgmap v180: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 212/1638 objects degraded (12.943%) 2024-03-30T14:32:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: 5 238932 10.40 MB/sec execute 457 sec latency 66.614 ms 2024-03-30T14:32:52.755 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:52 smithi031 ceph-mon[67456]: pgmap v181: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.0 MiB/s wr, 480 op/s; 221/1674 objects degraded (13.202%) 2024-03-30T14:32:52.755 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:52 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 221/1674 objects degraded (13.202%), 27 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:53.009 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:52 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:32:52.816+0000 7f7234c9a7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-30T14:32:53.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:52 smithi193 ceph-mon[52977]: pgmap v181: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.0 MiB/s wr, 480 op/s; 221/1674 objects degraded (13.202%) 2024-03-30T14:32:53.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:52 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 221/1674 objects degraded (13.202%), 27 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:53.170 INFO:teuthology.orchestra.run.smithi031.stdout: 5 239182 10.40 MB/sec execute 458 sec latency 46.760 ms 2024-03-30T14:32:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: 5 239461 10.39 MB/sec execute 459 sec latency 44.085 ms 2024-03-30T14:32:54.540 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:54 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:32:54.446+0000 7f7234c9a7c0 -1 osd.3 109 log_to_monitors true 2024-03-30T14:32:55.172 INFO:teuthology.orchestra.run.smithi031.stdout: 5 239925 10.38 MB/sec execute 460 sec latency 41.659 ms 2024-03-30T14:32:55.550 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:55 smithi031 ceph-mon[67456]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 509 op/s; 214/1632 objects degraded (13.113%) 2024-03-30T14:32:55.550 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:55 smithi031 ceph-mon[67456]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:32:55.550 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:55.551 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:55.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:55 smithi193 ceph-mon[52977]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 509 op/s; 214/1632 objects degraded (13.113%) 2024-03-30T14:32:55.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:55 smithi193 ceph-mon[52977]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:32:55.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:55.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:56.058 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:32:55 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:32:55.961+0000 7f72220ea700 -1 osd.3 109 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:32:56.175 INFO:teuthology.orchestra.run.smithi031.stdout: 5 240226 10.37 MB/sec execute 461 sec latency 95.635 ms 2024-03-30T14:32:56.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:56 smithi031 ceph-mon[67456]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T14:32:56.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:56 smithi031 ceph-mon[67456]: failed to encode map e112 with expected crc 2024-03-30T14:32:56.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:56 smithi031 ceph-mon[67456]: osdmap e112: 8 total, 7 up, 8 in 2024-03-30T14:32:56.565 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:56 smithi031 ceph-mon[67456]: failed to encode map e112 with expected crc 2024-03-30T14:32:56.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:56 smithi031 ceph-mon[67456]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:32:56.566 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:56 smithi031 ceph-mon[67456]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' 2024-03-30T14:32:56.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:56 smithi193 ceph-mon[52977]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T14:32:56.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:56 smithi193 ceph-mon[52977]: failed to encode map e112 with expected crc 2024-03-30T14:32:56.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:56 smithi193 ceph-mon[52977]: osdmap e112: 8 total, 7 up, 8 in 2024-03-30T14:32:56.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:56 smithi193 ceph-mon[52977]: failed to encode map e112 with expected crc 2024-03-30T14:32:56.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:56 smithi193 ceph-mon[52977]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-30T14:32:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:56 smithi193 ceph-mon[52977]: from='osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315]' entity='osd.3' 2024-03-30T14:32:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: 5 240663 10.39 MB/sec execute 462 sec latency 200.249 ms 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: pgmap v184: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 494 op/s; 219/1644 objects degraded (13.321%) 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315] boot 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.792 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:57 smithi031 ceph-mon[67456]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: pgmap v184: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 494 op/s; 219/1644 objects degraded (13.321%) 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: osd.3 [v2:172.21.15.31:6826/241147315,v1:172.21.15.31:6827/241147315] boot 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:32:57.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: failed to encode map e113 with expected crc 2024-03-30T14:32:57.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:57 smithi193 ceph-mon[52977]: failed to encode map e113 with expected crc 2024-03-30T14:32:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: 5 240663 10.37 MB/sec execute 463 sec latency 1200.292 ms 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: failed to encode map e113 with expected crc 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: failed to encode map e114 with expected crc 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: failed to encode map e114 with expected crc 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: failed to encode map e114 with expected crc 2024-03-30T14:32:58.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:58 smithi031 ceph-mon[67456]: failed to encode map e114 with expected crc 2024-03-30T14:32:59.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: failed to encode map e113 with expected crc 2024-03-30T14:32:59.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:32:59.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:59.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: failed to encode map e114 with expected crc 2024-03-30T14:32:59.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:32:59.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: failed to encode map e114 with expected crc 2024-03-30T14:32:59.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: failed to encode map e114 with expected crc 2024-03-30T14:32:59.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:58 smithi193 ceph-mon[52977]: failed to encode map e114 with expected crc 2024-03-30T14:32:59.175 INFO:teuthology.orchestra.run.smithi031.stdout: 5 240663 10.34 MB/sec execute 464 sec latency 2200.329 ms 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: failed to encode map e114 with expected crc 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: pgmap v187: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.9 MiB/s wr, 403 op/s; 225/1674 objects degraded (13.441%) 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 225/1674 objects degraded (13.441%), 27 pgs degraded (PG_DEGRADED) 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: failed to encode map e115 with expected crc 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: failed to encode map e115 with expected crc 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: failed to encode map e115 with expected crc 2024-03-30T14:32:59.656 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:32:59 smithi031 ceph-mon[67456]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: failed to encode map e114 with expected crc 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: pgmap v187: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.9 MiB/s wr, 403 op/s; 225/1674 objects degraded (13.441%) 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 225/1674 objects degraded (13.441%), 27 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: failed to encode map e115 with expected crc 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: failed to encode map e115 with expected crc 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: failed to encode map e115 with expected crc 2024-03-30T14:33:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:32:59 smithi193 ceph-mon[52977]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:33:00.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 240978 10.34 MB/sec execute 465 sec latency 2344.145 ms 2024-03-30T14:33:00.834 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:00 smithi031 ceph-mon[67456]: failed to encode map e115 with expected crc 2024-03-30T14:33:00.834 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:00 smithi031 ceph-mon[67456]: failed to encode map e116 with expected crc 2024-03-30T14:33:00.834 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:00 smithi031 ceph-mon[67456]: failed to encode map e116 with expected crc 2024-03-30T14:33:00.835 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:00 smithi031 ceph-mon[67456]: failed to encode map e116 with expected crc 2024-03-30T14:33:00.835 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:00 smithi031 ceph-mon[67456]: failed to encode map e116 with expected crc 2024-03-30T14:33:00.835 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:00 smithi031 ceph-mon[67456]: failed to encode map e116 with expected crc 2024-03-30T14:33:00.835 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:00 smithi031 ceph-mon[67456]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:33:01.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:00 smithi193 ceph-mon[52977]: failed to encode map e115 with expected crc 2024-03-30T14:33:01.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:00 smithi193 ceph-mon[52977]: failed to encode map e116 with expected crc 2024-03-30T14:33:01.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:00 smithi193 ceph-mon[52977]: failed to encode map e116 with expected crc 2024-03-30T14:33:01.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:00 smithi193 ceph-mon[52977]: failed to encode map e116 with expected crc 2024-03-30T14:33:01.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:00 smithi193 ceph-mon[52977]: failed to encode map e116 with expected crc 2024-03-30T14:33:01.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:00 smithi193 ceph-mon[52977]: failed to encode map e116 with expected crc 2024-03-30T14:33:01.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:00 smithi193 ceph-mon[52977]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:33:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 241245 10.34 MB/sec execute 466 sec latency 974.579 ms 2024-03-30T14:33:01.846 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: pgmap v190: 97 pgs: 2 peering, 1 active+recovering+degraded, 4 remapped+peering, 7 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+undersized+degraded, 67 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 4.1 MiB/s wr, 440 op/s; 142/1641 objects degraded (8.653%); 2.0 MiB/s, 10 objects/s recovering 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: failed to encode map e117 with expected crc 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: failed to encode map e117 with expected crc 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: failed to encode map e117 with expected crc 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: failed to encode map e117 with expected crc 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: failed to encode map e117 with expected crc 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:01.847 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:33:01.848 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:01.848 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:01.848 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:01.848 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:01.848 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:01 smithi031 ceph-mon[67456]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: pgmap v190: 97 pgs: 2 peering, 1 active+recovering+degraded, 4 remapped+peering, 7 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+undersized+degraded, 67 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 4.1 MiB/s wr, 440 op/s; 142/1641 objects degraded (8.653%); 2.0 MiB/s, 10 objects/s recovering 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: failed to encode map e117 with expected crc 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: failed to encode map e117 with expected crc 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: failed to encode map e117 with expected crc 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: failed to encode map e117 with expected crc 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: failed to encode map e117 with expected crc 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:02.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:01 smithi193 ceph-mon[52977]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:33:02.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 241561 10.33 MB/sec execute 467 sec latency 741.377 ms 2024-03-30T14:33:02.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: pgmap v192: 97 pgs: 4 peering, 1 active+recovering+degraded, 2 remapped+peering, 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 77 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.4 MiB/s wr, 458 op/s; 125/1629 objects degraded (7.673%); 2.5 MiB/s, 23 objects/s recovering 2024-03-30T14:33:02.799 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: pgmap v193: 97 pgs: 4 peering, 1 active+recovering+degraded, 2 remapped+peering, 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 77 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 520 op/s; 125/1629 objects degraded (7.673%); 2.9 MiB/s, 26 objects/s recovering 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: failed to encode map e118 with expected crc 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: failed to encode map e118 with expected crc 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: failed to encode map e118 with expected crc 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: failed to encode map e118 with expected crc 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: failed to encode map e118 with expected crc 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: failed to encode map e118 with expected crc 2024-03-30T14:33:02.800 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:02 smithi031 ceph-mon[67456]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:33:03.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: pgmap v192: 97 pgs: 4 peering, 1 active+recovering+degraded, 2 remapped+peering, 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 77 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.4 MiB/s wr, 458 op/s; 125/1629 objects degraded (7.673%); 2.5 MiB/s, 23 objects/s recovering 2024-03-30T14:33:03.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: pgmap v193: 97 pgs: 4 peering, 1 active+recovering+degraded, 2 remapped+peering, 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 77 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 520 op/s; 125/1629 objects degraded (7.673%); 2.9 MiB/s, 26 objects/s recovering 2024-03-30T14:33:03.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:03.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-30T14:33:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: failed to encode map e118 with expected crc 2024-03-30T14:33:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: failed to encode map e118 with expected crc 2024-03-30T14:33:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: failed to encode map e118 with expected crc 2024-03-30T14:33:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: failed to encode map e118 with expected crc 2024-03-30T14:33:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: failed to encode map e118 with expected crc 2024-03-30T14:33:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: failed to encode map e118 with expected crc 2024-03-30T14:33:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:02 smithi193 ceph-mon[52977]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:33:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 241733 10.32 MB/sec execute 468 sec latency 942.635 ms 2024-03-30T14:33:03.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:03 smithi031 ceph-mon[67456]: failed to encode map e119 with expected crc 2024-03-30T14:33:03.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:03 smithi031 ceph-mon[67456]: failed to encode map e119 with expected crc 2024-03-30T14:33:03.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:03 smithi031 ceph-mon[67456]: failed to encode map e119 with expected crc 2024-03-30T14:33:03.807 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:03 smithi031 ceph-mon[67456]: failed to encode map e119 with expected crc 2024-03-30T14:33:03.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:03 smithi031 ceph-mon[67456]: failed to encode map e119 with expected crc 2024-03-30T14:33:03.808 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:03 smithi031 ceph-mon[67456]: failed to encode map e119 with expected crc 2024-03-30T14:33:04.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:03 smithi193 ceph-mon[52977]: failed to encode map e119 with expected crc 2024-03-30T14:33:04.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:03 smithi193 ceph-mon[52977]: failed to encode map e119 with expected crc 2024-03-30T14:33:04.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:03 smithi193 ceph-mon[52977]: failed to encode map e119 with expected crc 2024-03-30T14:33:04.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:03 smithi193 ceph-mon[52977]: failed to encode map e119 with expected crc 2024-03-30T14:33:04.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:03 smithi193 ceph-mon[52977]: failed to encode map e119 with expected crc 2024-03-30T14:33:04.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:03 smithi193 ceph-mon[52977]: failed to encode map e119 with expected crc 2024-03-30T14:33:04.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 242020 10.31 MB/sec execute 469 sec latency 34.144 ms 2024-03-30T14:33:04.823 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:04 smithi031 ceph-mon[67456]: pgmap v196: 97 pgs: 7 peering, 1 active+recovering+degraded, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 83 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 716 B/s rd, 1.3 MiB/s wr, 176 op/s; 70/1593 objects degraded (4.394%); 1.3 MiB/s, 40 objects/s recovering 2024-03-30T14:33:04.823 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:04 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 70/1593 objects degraded (4.394%), 5 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:05.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:04 smithi193 ceph-mon[52977]: pgmap v196: 97 pgs: 7 peering, 1 active+recovering+degraded, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 83 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 716 B/s rd, 1.3 MiB/s wr, 176 op/s; 70/1593 objects degraded (4.394%); 1.3 MiB/s, 40 objects/s recovering 2024-03-30T14:33:05.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:04 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 70/1593 objects degraded (4.394%), 5 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:05.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 242401 10.31 MB/sec execute 470 sec latency 33.356 ms 2024-03-30T14:33:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 242726 10.31 MB/sec execute 471 sec latency 73.794 ms 2024-03-30T14:33:06.857 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:06 smithi031 ceph-mon[67456]: pgmap v197: 97 pgs: 5 peering, 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 88 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 2.8 MiB/s wr, 238 op/s; 58/1596 objects degraded (3.634%); 772 KiB/s, 28 objects/s recovering 2024-03-30T14:33:06.886 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:06 smithi193 ceph-mon[52977]: pgmap v197: 97 pgs: 5 peering, 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 88 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 2.8 MiB/s wr, 238 op/s; 58/1596 objects degraded (3.634%); 772 KiB/s, 28 objects/s recovering 2024-03-30T14:33:07.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 243005 10.30 MB/sec execute 472 sec latency 47.382 ms 2024-03-30T14:33:07.622 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:33:07.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:07 smithi031 ceph-mon[67456]: pgmap v198: 97 pgs: 97 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.8 MiB/s wr, 501 op/s; 1.4 MiB/s, 39 objects/s recovering 2024-03-30T14:33:07.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:07 smithi031 ceph-mon[67456]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:07.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:07 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/1596 objects degraded (3.634%), 3 pgs degraded) 2024-03-30T14:33:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:07 smithi193 ceph-mon[52977]: pgmap v198: 97 pgs: 97 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.8 MiB/s wr, 501 op/s; 1.4 MiB/s, 39 objects/s recovering 2024-03-30T14:33:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:07 smithi193 ceph-mon[52977]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:08.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:07 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/1596 objects degraded (3.634%), 3 pgs degraded) 2024-03-30T14:33:08.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 243225 10.30 MB/sec execute 473 sec latency 63.969 ms 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (7m) 13s ago 16m 23.7M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (3m) 13s ago 16m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (3m) 3m ago 15m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 13s ago 16m - - 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (10m) 13s ago 10m 40.4M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (10m) 3m ago 10m 14.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (6m) 13s ago 17m 507M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:33:08.520 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (5m) 3m ago 15m 449M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (4m) 13s ago 17m 52.1M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (4m) 3m ago 15m 36.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (10m) 13s ago 10m 280M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (10m) 3m ago 10m 82.5M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (6m) 13s ago 16m 17.3M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (6m) 3m ago 15m 15.3M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (2m) 13s ago 14m 363M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (118s) 13s ago 14m 303M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (71s) 13s ago 13m 227M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (23s) 13s ago 13m 12.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (13m) 3m ago 13m 546M 2704M 17.2.0 e1d6a67b021e 8f422c2830e6 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (12m) 3m ago 12m 402M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (12m) 3m ago 12m 357M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (11m) 3m ago 11m 453M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:33:08.521 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (5m) 13s ago 15m 70.0M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:33:08.909 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:08 smithi031 ceph-mon[67456]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:08.909 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:08 smithi031 ceph-mon[67456]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:09.053 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:33:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:33:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:33:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 4 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 8 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:33:09.054 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:33:09.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:08 smithi193 ceph-mon[52977]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:09.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:08 smithi193 ceph-mon[52977]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:09.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 243547 10.29 MB/sec execute 474 sec latency 61.318 ms 2024-03-30T14:33:09.453 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:33:09.453 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "10/23 daemons upgraded", 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:33:09.454 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:33:09.677 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:09 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/240611733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:09.677 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:09 smithi031 ceph-mon[67456]: pgmap v199: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 454 op/s; 1.1 MiB/s, 29 objects/s recovering 2024-03-30T14:33:09.677 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:09 smithi031 ceph-mon[67456]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:09.970 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:33:09.970 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:33:09.970 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:33:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:09 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/240611733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:09 smithi193 ceph-mon[52977]: pgmap v199: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 454 op/s; 1.1 MiB/s, 29 objects/s recovering 2024-03-30T14:33:10.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:09 smithi193 ceph-mon[52977]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 243852 10.28 MB/sec execute 475 sec latency 80.915 ms 2024-03-30T14:33:10.715 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:10 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/3441484590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:33:11.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:10 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/3441484590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:33:11.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 244045 10.27 MB/sec execute 476 sec latency 54.430 ms 2024-03-30T14:33:11.730 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:11 smithi031 ceph-mon[67456]: pgmap v200: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 395 op/s; 898 KiB/s, 24 objects/s recovering 2024-03-30T14:33:12.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:11 smithi193 ceph-mon[52977]: pgmap v200: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 395 op/s; 898 KiB/s, 24 objects/s recovering 2024-03-30T14:33:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 244374 10.28 MB/sec execute 477 sec latency 65.667 ms 2024-03-30T14:33:13.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 244722 10.27 MB/sec execute 478 sec latency 52.421 ms 2024-03-30T14:33:13.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:13.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:13.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:14.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:14.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 245122 10.27 MB/sec execute 479 sec latency 54.095 ms 2024-03-30T14:33:14.823 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:14 smithi031 ceph-mon[67456]: pgmap v201: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 347 op/s; 482 KiB/s, 14 objects/s recovering 2024-03-30T14:33:15.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:14 smithi193 ceph-mon[52977]: pgmap v201: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 347 op/s; 482 KiB/s, 14 objects/s recovering 2024-03-30T14:33:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 245344 10.26 MB/sec execute 480 sec latency 47.146 ms 2024-03-30T14:33:16.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 245587 10.26 MB/sec execute 481 sec latency 49.843 ms 2024-03-30T14:33:16.674 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:16 smithi031 ceph-mon[67456]: pgmap v202: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.7 MiB/s wr, 466 op/s; 429 KiB/s, 12 objects/s recovering 2024-03-30T14:33:16.674 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:16 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:17.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:16 smithi193 ceph-mon[52977]: pgmap v202: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.7 MiB/s wr, 466 op/s; 429 KiB/s, 12 objects/s recovering 2024-03-30T14:33:17.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:16 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 245913 10.26 MB/sec execute 482 sec latency 63.169 ms 2024-03-30T14:33:17.713 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:17 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:17.713 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:17 smithi193 ceph-mon[52977]: Upgrade: osd.4 is safe to restart 2024-03-30T14:33:17.723 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:17 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T14:33:17.723 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:17 smithi031 ceph-mon[67456]: Upgrade: osd.4 is safe to restart 2024-03-30T14:33:18.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 246220 10.25 MB/sec execute 483 sec latency 54.927 ms 2024-03-30T14:33:18.740 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:18 smithi031 ceph-mon[67456]: pgmap v203: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.6 MiB/s wr, 437 op/s; 423 KiB/s, 8 objects/s recovering 2024-03-30T14:33:19.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:18 smithi193 ceph-mon[52977]: pgmap v203: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.6 MiB/s wr, 437 op/s; 423 KiB/s, 8 objects/s recovering 2024-03-30T14:33:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 246481 10.24 MB/sec execute 484 sec latency 79.866 ms 2024-03-30T14:33:20.150 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:19 smithi031 ceph-mon[67456]: Upgrade: Updating osd.4 2024-03-30T14:33:20.150 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:20.151 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:33:20.151 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:20.151 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:19 smithi031 ceph-mon[67456]: Deploying daemon osd.4 on smithi193 2024-03-30T14:33:20.151 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:19 smithi031 ceph-mon[67456]: pgmap v204: 97 pgs: 97 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.2 MiB/s wr, 426 op/s 2024-03-30T14:33:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 246727 10.24 MB/sec execute 485 sec latency 50.308 ms 2024-03-30T14:33:20.282 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:19 smithi193 ceph-mon[52977]: Upgrade: Updating osd.4 2024-03-30T14:33:20.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:20.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:33:20.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:20.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:19 smithi193 ceph-mon[52977]: Deploying daemon osd.4 on smithi193 2024-03-30T14:33:20.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:19 smithi193 ceph-mon[52977]: pgmap v204: 97 pgs: 97 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.2 MiB/s wr, 426 op/s 2024-03-30T14:33:21.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 247064 10.23 MB/sec execute 486 sec latency 65.368 ms 2024-03-30T14:33:22.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 247382 10.22 MB/sec execute 487 sec latency 42.869 ms 2024-03-30T14:33:22.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:22 smithi193 ceph-mon[52977]: pgmap v205: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-30T14:33:22.464 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:22 smithi031 ceph-mon[67456]: pgmap v205: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-30T14:33:23.017 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:22 smithi193 systemd[1]: Stopping Ceph osd.4 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 247594 10.21 MB/sec execute 488 sec latency 45.946 ms 2024-03-30T14:33:23.291 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[31167]: 2024-03-30T14:33:23.064+0000 7f2b7d0f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:23.291 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[31167]: 2024-03-30T14:33:23.064+0000 7f2b7d0f9700 -1 osd.4 119 *** Got signal Terminated *** 2024-03-30T14:33:23.291 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[31167]: 2024-03-30T14:33:23.064+0000 7f2b7d0f9700 -1 osd.4 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:23.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:23 smithi193 ceph-mon[52977]: osd.4 marked itself down and dead 2024-03-30T14:33:23.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:23 smithi031 ceph-mon[67456]: osd.4 marked itself down and dead 2024-03-30T14:33:24.171 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:23 smithi193 podman[57497]: 2024-03-30 14:33:23.899332985 +0000 UTC m=+0.965349780 container died 8f422c2830e68179a327d81de184b6e29bee228784b0a30d373a2e46d32bf945 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.expose-services=, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-type=git, ceph=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_CLEAN=True, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-30T14:33:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 247645 10.19 MB/sec execute 489 sec latency 865.174 ms 2024-03-30T14:33:24.539 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:24 smithi193 ceph-mon[52977]: pgmap v206: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 452 op/s 2024-03-30T14:33:24.539 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:24 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:33:24.540 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:24 smithi193 ceph-mon[52977]: failed to encode map e120 with expected crc 2024-03-30T14:33:24.540 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:24 smithi193 ceph-mon[52977]: osdmap e120: 8 total, 7 up, 8 in 2024-03-30T14:33:24.540 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:24 smithi193 podman[57497]: 2024-03-30 14:33:24.27445562 +0000 UTC m=+1.340472415 container cleanup 8f422c2830e68179a327d81de184b6e29bee228784b0a30d373a2e46d32bf945 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, name=centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-type=git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, release=754, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, version=8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux ) 2024-03-30T14:33:24.540 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:24 smithi193 bash[57497]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T14:33:24.544 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:24 smithi031 ceph-mon[67456]: pgmap v206: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 452 op/s 2024-03-30T14:33:24.544 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:24 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:33:24.544 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:24 smithi031 ceph-mon[67456]: failed to encode map e120 with expected crc 2024-03-30T14:33:24.544 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:24 smithi031 ceph-mon[67456]: osdmap e120: 8 total, 7 up, 8 in 2024-03-30T14:33:24.834 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:24 smithi193 podman[57524]: 2024-03-30 14:33:24.542500766 +0000 UTC m=+0.643871173 container remove 8f422c2830e68179a327d81de184b6e29bee228784b0a30d373a2e46d32bf945 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git) 2024-03-30T14:33:25.126 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:24 smithi193 podman[57614]: 2024-03-30 14:33:24.774418608 +0000 UTC m=+0.018894763 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:25.126 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:24 smithi193 podman[57614]: 2024-03-30 14:33:24.893726883 +0000 UTC m=+0.138203017 container create 3237ed1927e326a01ab90449c5a8101be56f266ba490facb6305c2ca31f42ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:33:25.127 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:25 smithi193 podman[57614]: 2024-03-30 14:33:25.129856351 +0000 UTC m=+0.374332486 container init 3237ed1927e326a01ab90449c5a8101be56f266ba490facb6305c2ca31f42ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-03-30T14:33:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: 5 247899 10.20 MB/sec execute 490 sec latency 1052.284 ms 2024-03-30T14:33:25.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:25 smithi031 ceph-mon[67456]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.337 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:25 smithi031 ceph-mon[67456]: osdmap e121: 8 total, 7 up, 8 in 2024-03-30T14:33:25.338 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:25 smithi031 ceph-mon[67456]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.338 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:25 smithi031 ceph-mon[67456]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.338 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:25 smithi031 ceph-mon[67456]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.338 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:25 smithi031 ceph-mon[67456]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.338 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:25 smithi031 ceph-mon[67456]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:25 smithi193 ceph-mon[52977]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:25 smithi193 ceph-mon[52977]: osdmap e121: 8 total, 7 up, 8 in 2024-03-30T14:33:25.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:25 smithi193 ceph-mon[52977]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:25 smithi193 ceph-mon[52977]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:25 smithi193 ceph-mon[52977]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:25 smithi193 ceph-mon[52977]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:25 smithi193 ceph-mon[52977]: failed to encode map e121 with expected crc 2024-03-30T14:33:25.423 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:25 smithi193 podman[57614]: 2024-03-30 14:33:25.133311647 +0000 UTC m=+0.377787784 container start 3237ed1927e326a01ab90449c5a8101be56f266ba490facb6305c2ca31f42ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-30T14:33:25.423 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:25 smithi193 podman[57614]: 2024-03-30 14:33:25.221856339 +0000 UTC m=+0.466332492 container attach 3237ed1927e326a01ab90449c5a8101be56f266ba490facb6305c2ca31f42ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True) 2024-03-30T14:33:25.773 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:25 smithi193 podman[57614]: 2024-03-30 14:33:25.426227219 +0000 UTC m=+0.670703355 container died 3237ed1927e326a01ab90449c5a8101be56f266ba490facb6305c2ca31f42ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T14:33:26.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 248160 10.19 MB/sec execute 491 sec latency 83.364 ms 2024-03-30T14:33:26.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:26 smithi193 ceph-mon[52977]: pgmap v209: 97 pgs: 3 active+undersized, 4 active+undersized+degraded, 19 peering, 71 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.1 MiB/s wr, 444 op/s; 32/1653 objects degraded (1.936%) 2024-03-30T14:33:26.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:26 smithi193 ceph-mon[52977]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-30T14:33:26.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:26 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 32/1653 objects degraded (1.936%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:26.335 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 podman[57640]: 2024-03-30 14:33:26.067931358 +0000 UTC m=+0.635806444 container remove 3237ed1927e326a01ab90449c5a8101be56f266ba490facb6305c2ca31f42ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-4e50bbc) 2024-03-30T14:33:26.335 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.4.service: Deactivated successfully. 2024-03-30T14:33:26.335 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 systemd[1]: Stopped Ceph osd.4 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:33:26.335 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.4.service: Consumed 4min 35.217s CPU time. 2024-03-30T14:33:26.335 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 systemd[1]: Starting Ceph osd.4 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:26.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:26 smithi031 ceph-mon[67456]: pgmap v209: 97 pgs: 3 active+undersized, 4 active+undersized+degraded, 19 peering, 71 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.1 MiB/s wr, 444 op/s; 32/1653 objects degraded (1.936%) 2024-03-30T14:33:26.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:26 smithi031 ceph-mon[67456]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-30T14:33:26.344 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:26 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 32/1653 objects degraded (1.936%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:26.802 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 podman[57745]: 2024-03-30 14:33:26.41574042 +0000 UTC m=+0.018133045 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:26.802 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 podman[57745]: 2024-03-30 14:33:26.54415515 +0000 UTC m=+0.146547761 container create 8a61e400f09184988b73f4cf27cf5664977abff6899a0341bba303f8582c6cb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-30T14:33:26.803 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 podman[57745]: 2024-03-30 14:33:26.80561739 +0000 UTC m=+0.408010012 container init 8a61e400f09184988b73f4cf27cf5664977abff6899a0341bba303f8582c6cb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-30T14:33:27.084 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 podman[57745]: 2024-03-30 14:33:26.808842404 +0000 UTC m=+0.411235018 container start 8a61e400f09184988b73f4cf27cf5664977abff6899a0341bba303f8582c6cb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-30T14:33:27.084 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:26 smithi193 podman[57745]: 2024-03-30 14:33:26.939135601 +0000 UTC m=+0.541528214 container attach 8a61e400f09184988b73f4cf27cf5664977abff6899a0341bba303f8582c6cb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:33:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 248485 10.19 MB/sec execute 492 sec latency 66.477 ms 2024-03-30T14:33:27.584 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate[57760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T14:33:27.584 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate[57760]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T14:33:27.584 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 bash[57745]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T14:33:27.584 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 bash[57745]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T14:33:27.584 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 bash[57745]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 bash[57745]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 bash[57745]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 bash[57745]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 bash[57745]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate[57760]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate[57760]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate[57760]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate[57760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate[57760]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-30T14:33:27.585 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 podman[57745]: 2024-03-30 14:33:27.290234977 +0000 UTC m=+0.892627593 container died 8a61e400f09184988b73f4cf27cf5664977abff6899a0341bba303f8582c6cb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, ceph=True) 2024-03-30T14:33:28.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 248762 10.18 MB/sec execute 493 sec latency 58.142 ms 2024-03-30T14:33:28.205 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:27 smithi193 podman[57877]: 2024-03-30 14:33:27.952014754 +0000 UTC m=+0.656734724 container remove 8a61e400f09184988b73f4cf27cf5664977abff6899a0341bba303f8582c6cb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:33:28.205 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 podman[57942]: 2024-03-30 14:33:28.067595635 +0000 UTC m=+0.018375560 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:28.205 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 podman[57942]: 2024-03-30 14:33:28.194734262 +0000 UTC m=+0.145514212 container create a93871da45177d16974095b2978153df53adeac58107fa70b4c3a1af2e96a48b (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-30T14:33:28.502 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 podman[57942]: 2024-03-30 14:33:28.414077565 +0000 UTC m=+0.364857472 container init a93871da45177d16974095b2978153df53adeac58107fa70b4c3a1af2e96a48b (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc) 2024-03-30T14:33:28.502 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 podman[57942]: 2024-03-30 14:33:28.417338277 +0000 UTC m=+0.368118184 container start a93871da45177d16974095b2978153df53adeac58107fa70b4c3a1af2e96a48b (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-30T14:33:28.503 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 ceph-osd[57963]: -- 172.21.15.193:0/130940638 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55795f24f680 con 0x55795f218c00 2024-03-30T14:33:28.666 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:28 smithi031 ceph-mon[67456]: pgmap v210: 97 pgs: 9 active+undersized, 13 active+undersized+degraded, 19 peering, 56 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 389 op/s; 126/1629 objects degraded (7.735%) 2024-03-30T14:33:28.666 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:28.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:28.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:28.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:28.834 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 bash[57942]: a93871da45177d16974095b2978153df53adeac58107fa70b4c3a1af2e96a48b 2024-03-30T14:33:28.834 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 systemd[1]: Started Ceph osd.4 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:33:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:28 smithi193 ceph-mon[52977]: pgmap v210: 97 pgs: 9 active+undersized, 13 active+undersized+degraded, 19 peering, 56 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 389 op/s; 126/1629 objects degraded (7.735%) 2024-03-30T14:33:28.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:28.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:28.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:28.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:29.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 249005 10.17 MB/sec execute 494 sec latency 64.053 ms 2024-03-30T14:33:29.334 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:33:28.961+0000 7efc1f84b7c0 -1 Falling back to public interface 2024-03-30T14:33:29.682 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:33:29.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:33:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 249418 10.18 MB/sec execute 495 sec latency 48.552 ms 2024-03-30T14:33:30.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:30 smithi031 ceph-mon[67456]: pgmap v211: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.8 MiB/s wr, 395 op/s; 274/1617 objects degraded (16.945%) 2024-03-30T14:33:30.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:30 smithi031 ceph-mon[67456]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-03-30T14:33:31.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:30 smithi193 ceph-mon[52977]: pgmap v211: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.8 MiB/s wr, 395 op/s; 274/1617 objects degraded (16.945%) 2024-03-30T14:33:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:30 smithi193 ceph-mon[52977]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-03-30T14:33:31.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 249836 10.18 MB/sec execute 496 sec latency 43.814 ms 2024-03-30T14:33:31.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:31 smithi031 ceph-mon[67456]: pgmap v212: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.7 MiB/s wr, 313 op/s; 284/1692 objects degraded (16.785%) 2024-03-30T14:33:31.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:31 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 284/1692 objects degraded (16.785%), 28 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:32.012 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:31 smithi193 ceph-mon[52977]: pgmap v212: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.7 MiB/s wr, 313 op/s; 284/1692 objects degraded (16.785%) 2024-03-30T14:33:32.012 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:31 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 284/1692 objects degraded (16.785%), 28 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:32.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 250132 10.17 MB/sec execute 497 sec latency 41.549 ms 2024-03-30T14:33:33.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 250407 10.17 MB/sec execute 498 sec latency 59.006 ms 2024-03-30T14:33:33.770 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:33.770 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:33.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:33.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:34.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 250782 10.16 MB/sec execute 499 sec latency 69.634 ms 2024-03-30T14:33:34.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:34 smithi031 ceph-mon[67456]: pgmap v213: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 316 op/s; 286/1689 objects degraded (16.933%) 2024-03-30T14:33:34.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:34.797 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:34.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:34 smithi193 ceph-mon[52977]: pgmap v213: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 316 op/s; 286/1689 objects degraded (16.933%) 2024-03-30T14:33:34.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:34 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:34.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:34 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:35.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 251049 10.15 MB/sec execute 500 sec latency 63.488 ms 2024-03-30T14:33:35.731 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:35 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:33:35.412+0000 7efc1f84b7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-30T14:33:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 251263 10.14 MB/sec execute 501 sec latency 69.143 ms 2024-03-30T14:33:36.585 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:36 smithi031 ceph-mon[67456]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 465 op/s; 272/1689 objects degraded (16.104%) 2024-03-30T14:33:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:36 smithi193 ceph-mon[52977]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 465 op/s; 272/1689 objects degraded (16.104%) 2024-03-30T14:33:37.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 251562 10.16 MB/sec execute 502 sec latency 63.291 ms 2024-03-30T14:33:37.334 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:37 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:33:37.016+0000 7efc1f84b7c0 -1 osd.4 119 log_to_monitors true 2024-03-30T14:33:37.661 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:37 smithi031 ceph-mon[67456]: from='osd.4 [v2:172.21.15.193:6800/2182440575,v1:172.21.15.193:6801/2182440575]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:33:37.661 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:37 smithi031 ceph-mon[67456]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:33:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:37 smithi193 ceph-mon[52977]: from='osd.4 [v2:172.21.15.193:6800/2182440575,v1:172.21.15.193:6801/2182440575]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:33:37.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:37 smithi193 ceph-mon[52977]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:33:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 251791 10.14 MB/sec execute 503 sec latency 171.288 ms 2024-03-30T14:33:38.334 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:33:38.065+0000 7efc0cc9b700 -1 osd.4 119 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:33:38.666 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.6 MiB/s wr, 489 op/s; 269/1653 objects degraded (16.273%) 2024-03-30T14:33:38.666 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 269/1653 objects degraded (16.273%), 28 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: osdmap e122: 8 total, 7 up, 8 in 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: failed to encode map e122 with expected crc 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='osd.4 [v2:172.21.15.193:6800/2182440575,v1:172.21.15.193:6801/2182440575]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: failed to encode map e122 with expected crc 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: failed to encode map e122 with expected crc 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: failed to encode map e122 with expected crc 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: failed to encode map e122 with expected crc 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:33:38.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:38.668 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:38.668 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='osd.4 ' entity='osd.4' 2024-03-30T14:33:38.668 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:33:38.668 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:38.668 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:38.668 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:38.668 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:39.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.6 MiB/s wr, 489 op/s; 269/1653 objects degraded (16.273%) 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 269/1653 objects degraded (16.273%), 28 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: osdmap e122: 8 total, 7 up, 8 in 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: failed to encode map e122 with expected crc 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='osd.4 [v2:172.21.15.193:6800/2182440575,v1:172.21.15.193:6801/2182440575]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: failed to encode map e122 with expected crc 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: failed to encode map e122 with expected crc 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: failed to encode map e122 with expected crc 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: failed to encode map e122 with expected crc 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:39.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='osd.4 ' entity='osd.4' 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:39.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 252123 10.13 MB/sec execute 504 sec latency 101.574 ms 2024-03-30T14:33:39.860 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 485 op/s; 269/1653 objects degraded (16.273%) 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.0 MiB/s wr, 591 op/s; 269/1653 objects degraded (16.273%) 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: osd.4 [v2:172.21.15.193:6800/2182440575,v1:172.21.15.193:6801/2182440575] boot 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: failed to encode map e123 with expected crc 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: failed to encode map e123 with expected crc 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: failed to encode map e123 with expected crc 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: failed to encode map e123 with expected crc 2024-03-30T14:33:39.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:39 smithi031 ceph-mon[67456]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:33:40.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 485 op/s; 269/1653 objects degraded (16.273%) 2024-03-30T14:33:40.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.0 MiB/s wr, 591 op/s; 269/1653 objects degraded (16.273%) 2024-03-30T14:33:40.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:40.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-30T14:33:40.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: osd.4 [v2:172.21.15.193:6800/2182440575,v1:172.21.15.193:6801/2182440575] boot 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: failed to encode map e123 with expected crc 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: failed to encode map e123 with expected crc 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: failed to encode map e123 with expected crc 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: failed to encode map e123 with expected crc 2024-03-30T14:33:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:39 smithi193 ceph-mon[52977]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:33:40.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 252260 10.12 MB/sec execute 505 sec latency 1101.616 ms 2024-03-30T14:33:40.390 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: failed to encode map e124 with expected crc 2024-03-30T14:33:40.664 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:40 smithi031 ceph-mon[67456]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:33:41.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: failed to encode map e124 with expected crc 2024-03-30T14:33:41.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:40 smithi193 ceph-mon[52977]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:33:41.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 252415 10.11 MB/sec execute 506 sec latency 1517.321 ms 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (7m) 46s ago 16m 23.7M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (3m) 46s ago 16m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (3m) 8s ago 16m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 46s ago 16m - - 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (11m) 46s ago 11m 40.4M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (10m) 8s ago 10m 15.7M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (6m) 46s ago 18m 507M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (5m) 8s ago 16m 452M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (5m) 46s ago 18m 52.1M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (4m) 8s ago 16m 41.9M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (10m) 46s ago 10m 280M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:33:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (10m) 8s ago 10m 85.1M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (7m) 46s ago 16m 17.3M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (7m) 8s ago 16m 15.0M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (3m) 46s ago 15m 363M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (2m) 46s ago 14m 303M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (104s) 46s ago 14m 227M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (56s) 46s ago 14m 12.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (13s) 8s ago 13m 12.5M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (13m) 8s ago 13m 593M 2704M 17.2.0 e1d6a67b021e 77bd5f26946d 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (12m) 8s ago 12m 554M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (12m) 8s ago 12m 760M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:33:41.373 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (5m) 46s ago 15m 70.0M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:33:41.736 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:41 smithi031 ceph-mon[67456]: pgmap v221: 97 pgs: 1 remapped+peering, 5 activating, 11 active+undersized, 24 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 3.4 MiB/s wr, 174 op/s; 247/1596 objects degraded (15.476%) 2024-03-30T14:33:41.736 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:41 smithi031 ceph-mon[67456]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:41.736 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:41 smithi031 ceph-mon[67456]: failed to encode map e125 with expected crc 2024-03-30T14:33:41.736 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:41 smithi031 ceph-mon[67456]: failed to encode map e125 with expected crc 2024-03-30T14:33:41.736 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:41 smithi031 ceph-mon[67456]: failed to encode map e125 with expected crc 2024-03-30T14:33:41.736 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:41 smithi031 ceph-mon[67456]: failed to encode map e125 with expected crc 2024-03-30T14:33:42.037 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:33:42.037 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:33:42.037 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:33:42.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:42.037 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:33:42.037 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:33:42.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 5 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 9 2024-03-30T14:33:42.038 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:33:42.039 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:33:42.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:41 smithi193 ceph-mon[52977]: pgmap v221: 97 pgs: 1 remapped+peering, 5 activating, 11 active+undersized, 24 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 3.4 MiB/s wr, 174 op/s; 247/1596 objects degraded (15.476%) 2024-03-30T14:33:42.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:41 smithi193 ceph-mon[52977]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:42.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:41 smithi193 ceph-mon[52977]: failed to encode map e125 with expected crc 2024-03-30T14:33:42.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:41 smithi193 ceph-mon[52977]: failed to encode map e125 with expected crc 2024-03-30T14:33:42.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:41 smithi193 ceph-mon[52977]: failed to encode map e125 with expected crc 2024-03-30T14:33:42.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:41 smithi193 ceph-mon[52977]: failed to encode map e125 with expected crc 2024-03-30T14:33:42.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 252584 10.10 MB/sec execute 507 sec latency 45.732 ms 2024-03-30T14:33:42.466 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:33:42.466 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:33:42.466 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:33:42.466 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "11/23 daemons upgraded", 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:33:42.467 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: failed to encode map e126 with expected crc 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: failed to encode map e126 with expected crc 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: failed to encode map e126 with expected crc 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: failed to encode map e126 with expected crc 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2249946283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:42.766 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:42.977 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 209/1557 objects degraded (13.423%), 19 pgs degraded 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 209/1557 objects degraded (13.423%), 19 pgs degraded 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.b is active+undersized+degraded, acting [7,5] 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.18 is active+undersized+degraded, acting [5,2] 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1 is active+recovery_wait+undersized+degraded+remapped, acting [0,3] 2024-03-30T14:33:42.978 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.6 is active+recovery_wait+undersized+degraded+remapped, acting [0,1] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-03-30T14:33:42.979 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-03-30T14:33:43.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:43.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:43.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: failed to encode map e126 with expected crc 2024-03-30T14:33:43.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:33:43.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: failed to encode map e126 with expected crc 2024-03-30T14:33:43.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: failed to encode map e126 with expected crc 2024-03-30T14:33:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: failed to encode map e126 with expected crc 2024-03-30T14:33:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2249946283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:33:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:43.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 252802 10.09 MB/sec execute 508 sec latency 555.057 ms 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: pgmap v224: 97 pgs: 1 remapped+peering, 5 activating, 9 active+undersized, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 62 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.0 MiB/s wr, 325 op/s; 209/1557 objects degraded (13.423%); 158 KiB/s, 15 objects/s recovering 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 209/1557 objects degraded (13.423%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: failed to encode map e127 with expected crc 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: failed to encode map e127 with expected crc 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: failed to encode map e127 with expected crc 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: failed to encode map e127 with expected crc 2024-03-30T14:33:43.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: failed to encode map e127 with expected crc 2024-03-30T14:33:43.783 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:33:43.783 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: failed to encode map e127 with expected crc 2024-03-30T14:33:43.783 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: failed to encode map e127 with expected crc 2024-03-30T14:33:43.783 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:43 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/617252377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:33:44.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: pgmap v224: 97 pgs: 1 remapped+peering, 5 activating, 9 active+undersized, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 62 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.0 MiB/s wr, 325 op/s; 209/1557 objects degraded (13.423%); 158 KiB/s, 15 objects/s recovering 2024-03-30T14:33:44.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 209/1557 objects degraded (13.423%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: failed to encode map e127 with expected crc 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: failed to encode map e127 with expected crc 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: failed to encode map e127 with expected crc 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: failed to encode map e127 with expected crc 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: failed to encode map e127 with expected crc 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: failed to encode map e127 with expected crc 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: failed to encode map e127 with expected crc 2024-03-30T14:33:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:43 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/617252377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:33:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 253102 10.08 MB/sec execute 509 sec latency 1029.311 ms 2024-03-30T14:33:44.818 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:44 smithi031 ceph-mon[67456]: failed to encode map e128 with expected crc 2024-03-30T14:33:44.818 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:44 smithi031 ceph-mon[67456]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:33:44.819 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:44 smithi031 ceph-mon[67456]: failed to encode map e128 with expected crc 2024-03-30T14:33:44.819 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:44 smithi031 ceph-mon[67456]: failed to encode map e128 with expected crc 2024-03-30T14:33:44.819 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:44 smithi031 ceph-mon[67456]: failed to encode map e128 with expected crc 2024-03-30T14:33:45.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:44 smithi193 ceph-mon[52977]: failed to encode map e128 with expected crc 2024-03-30T14:33:45.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:44 smithi193 ceph-mon[52977]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:33:45.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:44 smithi193 ceph-mon[52977]: failed to encode map e128 with expected crc 2024-03-30T14:33:45.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:44 smithi193 ceph-mon[52977]: failed to encode map e128 with expected crc 2024-03-30T14:33:45.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:44 smithi193 ceph-mon[52977]: failed to encode map e128 with expected crc 2024-03-30T14:33:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 253506 10.09 MB/sec execute 510 sec latency 48.320 ms 2024-03-30T14:33:45.853 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:45 smithi031 ceph-mon[67456]: pgmap v227: 97 pgs: 5 peering, 5 activating, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 472 op/s; 65/1578 objects degraded (4.119%); 1.2 MiB/s, 57 objects/s recovering 2024-03-30T14:33:46.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:45 smithi193 ceph-mon[52977]: pgmap v227: 97 pgs: 5 peering, 5 activating, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 472 op/s; 65/1578 objects degraded (4.119%); 1.2 MiB/s, 57 objects/s recovering 2024-03-30T14:33:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: 5 253893 10.10 MB/sec execute 511 sec latency 54.432 ms 2024-03-30T14:33:46.868 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:46 smithi031 ceph-mon[67456]: pgmap v228: 97 pgs: 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 363 op/s; 43/1608 objects degraded (2.674%); 1.1 MiB/s, 44 keys/s, 42 objects/s recovering 2024-03-30T14:33:47.039 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:46 smithi193 ceph-mon[52977]: pgmap v228: 97 pgs: 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 363 op/s; 43/1608 objects degraded (2.674%); 1.1 MiB/s, 44 keys/s, 42 objects/s recovering 2024-03-30T14:33:47.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 254329 10.09 MB/sec execute 512 sec latency 37.517 ms 2024-03-30T14:33:48.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 254866 10.10 MB/sec execute 513 sec latency 26.165 ms 2024-03-30T14:33:48.378 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:48 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/1608 objects degraded (2.674%), 2 pgs degraded) 2024-03-30T14:33:48.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:48 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/1608 objects degraded (2.674%), 2 pgs degraded) 2024-03-30T14:33:49.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 255541 10.12 MB/sec execute 514 sec latency 35.088 ms 2024-03-30T14:33:49.394 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:49 smithi031 ceph-mon[67456]: pgmap v229: 97 pgs: 3 peering, 94 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.7 MiB/s wr, 522 op/s; 1.2 MiB/s, 38 keys/s, 47 objects/s recovering 2024-03-30T14:33:49.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:49 smithi193 ceph-mon[52977]: pgmap v229: 97 pgs: 3 peering, 94 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.7 MiB/s wr, 522 op/s; 1.2 MiB/s, 38 keys/s, 47 objects/s recovering 2024-03-30T14:33:50.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 255847 10.12 MB/sec execute 515 sec latency 68.586 ms 2024-03-30T14:33:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 256136 10.11 MB/sec execute 516 sec latency 54.593 ms 2024-03-30T14:33:51.180 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:51 smithi031 ceph-mon[67456]: pgmap v230: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 433 op/s; 972 KiB/s, 30 keys/s, 32 objects/s recovering 2024-03-30T14:33:51.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:51 smithi193 ceph-mon[52977]: pgmap v230: 97 pgs: 97 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 433 op/s; 972 KiB/s, 30 keys/s, 32 objects/s recovering 2024-03-30T14:33:52.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 256421 10.10 MB/sec execute 517 sec latency 65.315 ms 2024-03-30T14:33:53.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 256759 10.10 MB/sec execute 518 sec latency 43.457 ms 2024-03-30T14:33:53.292 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:53 smithi031 ceph-mon[67456]: pgmap v231: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.0 MiB/s wr, 438 op/s; 825 KiB/s, 25 keys/s, 27 objects/s recovering 2024-03-30T14:33:53.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:53 smithi193 ceph-mon[52977]: pgmap v231: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.0 MiB/s wr, 438 op/s; 825 KiB/s, 25 keys/s, 27 objects/s recovering 2024-03-30T14:33:54.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 256955 10.09 MB/sec execute 519 sec latency 76.431 ms 2024-03-30T14:33:54.348 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:54.349 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:54 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:54.349 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:54 smithi031 ceph-mon[67456]: Upgrade: osd.5 is safe to restart 2024-03-30T14:33:54.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:54.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:54 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T14:33:54.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:54 smithi193 ceph-mon[52977]: Upgrade: osd.5 is safe to restart 2024-03-30T14:33:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 257164 10.09 MB/sec execute 520 sec latency 58.911 ms 2024-03-30T14:33:55.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:55 smithi193 ceph-mon[52977]: pgmap v232: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.5 MiB/s wr, 549 op/s; 321 KiB/s, 23 keys/s, 8 objects/s recovering 2024-03-30T14:33:55.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:55.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:33:55.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:55.373 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:55 smithi031 ceph-mon[67456]: pgmap v232: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.5 MiB/s wr, 549 op/s; 321 KiB/s, 23 keys/s, 8 objects/s recovering 2024-03-30T14:33:55.373 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:55.374 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:33:55.374 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:33:56.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 257398 10.09 MB/sec execute 521 sec latency 75.658 ms 2024-03-30T14:33:56.419 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:56 smithi031 ceph-mon[67456]: Upgrade: Updating osd.5 2024-03-30T14:33:56.419 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:56 smithi031 ceph-mon[67456]: Deploying daemon osd.5 on smithi193 2024-03-30T14:33:56.511 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:56 smithi193 ceph-mon[52977]: Upgrade: Updating osd.5 2024-03-30T14:33:56.512 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:56 smithi193 ceph-mon[52977]: Deploying daemon osd.5 on smithi193 2024-03-30T14:33:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 257666 10.08 MB/sec execute 522 sec latency 76.559 ms 2024-03-30T14:33:57.180 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:57 smithi031 ceph-mon[67456]: pgmap v233: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 496 op/s; 279 KiB/s, 20 keys/s, 7 objects/s recovering 2024-03-30T14:33:57.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:57 smithi193 ceph-mon[52977]: pgmap v233: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 496 op/s; 279 KiB/s, 20 keys/s, 7 objects/s recovering 2024-03-30T14:33:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: 5 258045 10.07 MB/sec execute 523 sec latency 40.251 ms 2024-03-30T14:33:58.762 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:58.763 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:58.763 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:58.782 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:58.782 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:58.782 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:33:59.084 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:33:58 smithi193 systemd[1]: Stopping Ceph osd.5 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:59.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 258299 10.07 MB/sec execute 524 sec latency 55.382 ms 2024-03-30T14:33:59.584 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:33:59 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[34139]: 2024-03-30T14:33:59.132+0000 7f2add52a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:59.584 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:33:59 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[34139]: 2024-03-30T14:33:59.132+0000 7f2add52a700 -1 osd.5 128 *** Got signal Terminated *** 2024-03-30T14:33:59.584 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:33:59 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[34139]: 2024-03-30T14:33:59.132+0000 7f2add52a700 -1 osd.5 128 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:59.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:59 smithi031 ceph-mon[67456]: pgmap v234: 97 pgs: 97 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 495 op/s; 173 KiB/s, 5 objects/s recovering 2024-03-30T14:33:59.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:33:59 smithi031 ceph-mon[67456]: osd.5 marked itself down and dead 2024-03-30T14:34:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:59 smithi193 ceph-mon[52977]: pgmap v234: 97 pgs: 97 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 495 op/s; 173 KiB/s, 5 objects/s recovering 2024-03-30T14:34:00.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:33:59 smithi193 ceph-mon[52977]: osd.5 marked itself down and dead 2024-03-30T14:34:00.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 258418 10.05 MB/sec execute 525 sec latency 526.268 ms 2024-03-30T14:34:00.584 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:00 smithi193 podman[59748]: 2024-03-30 14:34:00.323629545 +0000 UTC m=+1.315731817 container died 77bd5f26946d5a691e7920bc1fdd9c05d3642963e7a432c1423c3be35132c701 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, version=8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64) 2024-03-30T14:34:00.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:00 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:34:00.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:00 smithi031 ceph-mon[67456]: osdmap e129: 8 total, 7 up, 8 in 2024-03-30T14:34:00.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:00 smithi031 ceph-mon[67456]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:00 smithi031 ceph-mon[67456]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:00 smithi031 ceph-mon[67456]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:00 smithi031 ceph-mon[67456]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:00 smithi031 ceph-mon[67456]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.887 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:00 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:34:00.887 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:00 smithi193 ceph-mon[52977]: osdmap e129: 8 total, 7 up, 8 in 2024-03-30T14:34:00.887 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:00 smithi193 ceph-mon[52977]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.887 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:00 smithi193 ceph-mon[52977]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.887 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:00 smithi193 ceph-mon[52977]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.887 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:00 smithi193 ceph-mon[52977]: failed to encode map e129 with expected crc 2024-03-30T14:34:00.887 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:00 smithi193 ceph-mon[52977]: failed to encode map e129 with expected crc 2024-03-30T14:34:01.154 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:00 smithi193 podman[59748]: 2024-03-30 14:34:00.889841775 +0000 UTC m=+1.881944047 container cleanup 77bd5f26946d5a691e7920bc1fdd9c05d3642963e7a432c1423c3be35132c701 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, name=centos-stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True, vendor=Red Hat, Inc.) 2024-03-30T14:34:01.155 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:00 smithi193 bash[59748]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T14:34:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 258633 10.05 MB/sec execute 526 sec latency 1047.780 ms 2024-03-30T14:34:01.548 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:01 smithi193 podman[59775]: 2024-03-30 14:34:01.158072116 +0000 UTC m=+0.838064281 container remove 77bd5f26946d5a691e7920bc1fdd9c05d3642963e7a432c1423c3be35132c701 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, ceph=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vcs-type=git, maintainer=Guillaume Abrioux , distribution-scope=public, version=8, io.k8s.display-name=CentOS Stream 8) 2024-03-30T14:34:01.605 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: pgmap v236: 97 pgs: 12 stale+active+clean, 85 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: osdmap e130: 8 total, 7 up, 8 in 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.606 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:01 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.802 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:01 smithi193 podman[59862]: 2024-03-30 14:34:01.455029556 +0000 UTC m=+0.022643758 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:01.802 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:01 smithi193 podman[59862]: 2024-03-30 14:34:01.575820679 +0000 UTC m=+0.143434854 container create 12b458bad4f6c251097b500037539dd2b4720b2638313962b362ac981891eb00 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:01.802 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:01 smithi193 podman[59862]: 2024-03-30 14:34:01.805050155 +0000 UTC m=+0.372664332 container init 12b458bad4f6c251097b500037539dd2b4720b2638313962b362ac981891eb00 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:01.802 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: pgmap v236: 97 pgs: 12 stale+active+clean, 85 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-03-30T14:34:01.802 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.802 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.802 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: osdmap e130: 8 total, 7 up, 8 in 2024-03-30T14:34:01.802 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.803 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.803 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.803 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.803 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.803 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:01.803 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:01 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:02.084 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:01 smithi193 podman[59862]: 2024-03-30 14:34:01.809238604 +0000 UTC m=+0.376852785 container start 12b458bad4f6c251097b500037539dd2b4720b2638313962b362ac981891eb00 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-30T14:34:02.084 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:01 smithi193 podman[59862]: 2024-03-30 14:34:01.912813738 +0000 UTC m=+0.480427922 container attach 12b458bad4f6c251097b500037539dd2b4720b2638313962b362ac981891eb00 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-30T14:34:02.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 258980 10.05 MB/sec execute 527 sec latency 48.585 ms 2024-03-30T14:34:02.472 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:02 smithi193 podman[59862]: 2024-03-30 14:34:02.151186816 +0000 UTC m=+0.718801008 container died 12b458bad4f6c251097b500037539dd2b4720b2638313962b362ac981891eb00 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-30T14:34:02.621 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:02 smithi031 ceph-mon[67456]: failed to encode map e130 with expected crc 2024-03-30T14:34:02.756 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:02 smithi193 ceph-mon[52977]: failed to encode map e130 with expected crc 2024-03-30T14:34:03.006 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:02 smithi193 podman[59920]: 2024-03-30 14:34:02.75901027 +0000 UTC m=+0.596985009 container remove 12b458bad4f6c251097b500037539dd2b4720b2638313962b362ac981891eb00 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-30T14:34:03.006 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:02 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.5.service: Deactivated successfully. 2024-03-30T14:34:03.006 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:02 smithi193 systemd[1]: Stopped Ceph osd.5 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:03.006 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:02 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.5.service: Consumed 5min 16.427s CPU time. 2024-03-30T14:34:03.006 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:03 smithi193 systemd[1]: Starting Ceph osd.5 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 259317 10.05 MB/sec execute 528 sec latency 41.697 ms 2024-03-30T14:34:03.282 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:03 smithi193 podman[60024]: 2024-03-30 14:34:03.141870016 +0000 UTC m=+0.020454383 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:03.585 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:03 smithi193 podman[60024]: 2024-03-30 14:34:03.285533777 +0000 UTC m=+0.164118125 container create 7c07401eb3c2f8167a794df5ca45d80b4894220ab82766344fe3e26ee3309bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:03.585 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:03 smithi193 podman[60024]: 2024-03-30 14:34:03.505139143 +0000 UTC m=+0.383723506 container init 7c07401eb3c2f8167a794df5ca45d80b4894220ab82766344fe3e26ee3309bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-30T14:34:03.585 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:03 smithi193 podman[60024]: 2024-03-30 14:34:03.509498433 +0000 UTC m=+0.388082783 container start 7c07401eb3c2f8167a794df5ca45d80b4894220ab82766344fe3e26ee3309bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-30T14:34:03.642 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:03 smithi031 ceph-mon[67456]: pgmap v238: 97 pgs: 1 active+undersized, 4 active+undersized+degraded, 6 peering, 8 stale+active+clean, 78 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.5 MiB/s wr, 293 op/s; 26/1638 objects degraded (1.587%) 2024-03-30T14:34:03.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:03 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 26/1638 objects degraded (1.587%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:04.024 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:03 smithi193 ceph-mon[52977]: pgmap v238: 97 pgs: 1 active+undersized, 4 active+undersized+degraded, 6 peering, 8 stale+active+clean, 78 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.5 MiB/s wr, 293 op/s; 26/1638 objects degraded (1.587%) 2024-03-30T14:34:04.024 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:03 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 26/1638 objects degraded (1.587%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:04.024 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:03 smithi193 podman[60024]: 2024-03-30 14:34:03.605274537 +0000 UTC m=+0.483858914 container attach 7c07401eb3c2f8167a794df5ca45d80b4894220ab82766344fe3e26ee3309bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc) 2024-03-30T14:34:04.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 259599 10.04 MB/sec execute 529 sec latency 41.317 ms 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate[60039]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 bash[60024]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 bash[60024]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 bash[60024]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 bash[60024]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 bash[60024]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 bash[60024]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 bash[60024]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate[60039]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-30T14:34:04.335 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate[60039]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-30T14:34:04.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate[60039]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-30T14:34:04.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate[60039]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-30T14:34:04.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate[60039]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T14:34:04.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate[60039]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-30T14:34:04.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 podman[60024]: 2024-03-30 14:34:04.058981199 +0000 UTC m=+0.937565554 container died 7c07401eb3c2f8167a794df5ca45d80b4894220ab82766344fe3e26ee3309bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T14:34:04.900 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:04 smithi031 ceph-mon[67456]: pgmap v239: 97 pgs: 5 active+undersized, 15 active+undersized+degraded, 6 peering, 1 stale+active+clean, 70 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 458 op/s; 114/1638 objects degraded (6.960%) 2024-03-30T14:34:04.900 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:04 smithi031 ceph-mon[67456]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:34:05.025 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-mon[52977]: pgmap v239: 97 pgs: 5 active+undersized, 15 active+undersized+degraded, 6 peering, 1 stale+active+clean, 70 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 458 op/s; 114/1638 objects degraded (6.960%) 2024-03-30T14:34:05.025 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:04 smithi193 ceph-mon[52977]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:34:05.025 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 podman[60171]: 2024-03-30 14:34:04.651919231 +0000 UTC m=+0.587217583 container remove 7c07401eb3c2f8167a794df5ca45d80b4894220ab82766344fe3e26ee3309bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-03-30T14:34:05.025 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:04 smithi193 podman[60233]: 2024-03-30 14:34:04.772512204 +0000 UTC m=+0.018812070 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:05.025 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:05 smithi193 podman[60233]: 2024-03-30 14:34:05.02809313 +0000 UTC m=+0.274392979 container create faf1b49d03f17ccb79f7d0e9dd4d32fa119d61f612ee55b57def6cda5c777394 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-30T14:34:05.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 259942 10.04 MB/sec execute 530 sec latency 57.723 ms 2024-03-30T14:34:05.296 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:05 smithi193 podman[60233]: 2024-03-30 14:34:05.272726448 +0000 UTC m=+0.519026307 container init faf1b49d03f17ccb79f7d0e9dd4d32fa119d61f612ee55b57def6cda5c777394 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc) 2024-03-30T14:34:05.296 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:05 smithi193 podman[60233]: 2024-03-30 14:34:05.276139065 +0000 UTC m=+0.522438924 container start faf1b49d03f17ccb79f7d0e9dd4d32fa119d61f612ee55b57def6cda5c777394 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:34:05.584 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:05 smithi193 ceph-osd[60251]: -- 172.21.15.193:0/2269540645 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e35f6e5680 con 0x55e35f6aec00 2024-03-30T14:34:05.584 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:05 smithi193 bash[60233]: faf1b49d03f17ccb79f7d0e9dd4d32fa119d61f612ee55b57def6cda5c777394 2024-03-30T14:34:05.584 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:05 smithi193 systemd[1]: Started Ceph osd.5 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:06.084 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:05 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:34:05.825+0000 7f3a5a61c7c0 -1 Falling back to public interface 2024-03-30T14:34:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 260339 10.04 MB/sec execute 531 sec latency 51.316 ms 2024-03-30T14:34:06.469 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:06.469 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:06.469 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:06 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:06.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:06.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:06.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:06 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:07.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 260687 10.04 MB/sec execute 532 sec latency 42.388 ms 2024-03-30T14:34:07.472 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:07 smithi031 ceph-mon[67456]: pgmap v240: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 493 op/s; 179/1647 objects degraded (10.868%) 2024-03-30T14:34:07.473 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:07 smithi031 ceph-mon[67456]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-30T14:34:07.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:07 smithi193 ceph-mon[52977]: pgmap v240: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 493 op/s; 179/1647 objects degraded (10.868%) 2024-03-30T14:34:07.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:07 smithi193 ceph-mon[52977]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-30T14:34:08.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 261068 10.04 MB/sec execute 533 sec latency 48.905 ms 2024-03-30T14:34:08.493 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:08 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 182/1653 objects degraded (11.010%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:08.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:08 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 182/1653 objects degraded (11.010%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:09.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 261402 10.03 MB/sec execute 534 sec latency 56.910 ms 2024-03-30T14:34:09.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:09 smithi031 ceph-mon[67456]: pgmap v241: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.6 MiB/s wr, 317 op/s; 182/1653 objects degraded (11.010%) 2024-03-30T14:34:09.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:09 smithi193 ceph-mon[52977]: pgmap v241: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.6 MiB/s wr, 317 op/s; 182/1653 objects degraded (11.010%) 2024-03-30T14:34:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 261680 10.03 MB/sec execute 535 sec latency 62.830 ms 2024-03-30T14:34:10.765 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:10 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:10.765 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:10 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:10.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:10 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:10.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:10 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:11.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 261935 10.02 MB/sec execute 536 sec latency 54.984 ms 2024-03-30T14:34:11.772 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:11 smithi031 ceph-mon[67456]: pgmap v242: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 382 op/s; 182/1632 objects degraded (11.152%) 2024-03-30T14:34:11.773 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:11 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:11.773 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:11 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:11.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:11 smithi193 ceph-mon[52977]: pgmap v242: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 382 op/s; 182/1632 objects degraded (11.152%) 2024-03-30T14:34:11.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:11 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:11.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:11 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: 5 262247 10.03 MB/sec execute 537 sec latency 68.015 ms 2024-03-30T14:34:13.179 INFO:teuthology.orchestra.run.smithi031.stdout: 5 262589 10.02 MB/sec execute 538 sec latency 91.561 ms 2024-03-30T14:34:13.464 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:34:13.787 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:13 smithi031 ceph-mon[67456]: pgmap v243: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 440 op/s; 179/1644 objects degraded (10.888%) 2024-03-30T14:34:13.788 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:13.788 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:13.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:13 smithi193 ceph-mon[52977]: pgmap v243: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 440 op/s; 179/1644 objects degraded (10.888%) 2024-03-30T14:34:13.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:13.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:14.182 INFO:teuthology.orchestra.run.smithi031.stdout: 5 262843 10.02 MB/sec execute 539 sec latency 62.698 ms 2024-03-30T14:34:14.334 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:14 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:34:14.061+0000 7f3a5a61c7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-30T14:34:14.355 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:34:14.355 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (8m) 79s ago 17m 23.7M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:34:14.355 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (4m) 79s ago 17m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:34:14.355 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (4m) 4s ago 16m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:34:14.355 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 79s ago 17m - - 2024-03-30T14:34:14.355 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (11m) 79s ago 11m 40.4M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (11m) 4s ago 11m 15.8M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (7m) 79s ago 18m 507M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (6m) 4s ago 16m 452M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (6m) 79s ago 18m 52.1M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (5m) 4s ago 16m 43.5M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (11m) 79s ago 11m 280M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (11m) 4s ago 11m 85.1M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (8m) 79s ago 17m 17.3M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (7m) 4s ago 16m 15.1M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (3m) 79s ago 16m 363M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (3m) 79s ago 15m 303M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (2m) 79s ago 15m 227M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (89s) 79s ago 14m 12.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (46s) 4s ago 14m 274M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:34:14.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (9s) 4s ago 13m 11.9M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:34:14.357 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (13m) 4s ago 13m 586M 2704M 17.2.0 e1d6a67b021e 87eed05f7b35 2024-03-30T14:34:14.357 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (13m) 4s ago 13m 797M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:34:14.357 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (6m) 79s ago 16m 70.0M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:34:14.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:14 smithi031 ceph-mon[67456]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:14.851 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:14 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 179/1644 objects degraded (10.888%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:14.910 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:34:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:34:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 5 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 9 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:34:14.911 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:34:15.024 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:14 smithi193 ceph-mon[52977]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:15.025 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:14 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 179/1644 objects degraded (10.888%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: 5 263083 10.01 MB/sec execute 540 sec latency 47.681 ms 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "12/23 daemons upgraded", 2024-03-30T14:34:15.361 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:34:15.362 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:34:15.362 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: pgmap v244: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 475 op/s; 180/1650 objects degraded (10.909%) 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1335921513' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:15.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:15.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:15.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:15.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:15.962 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 180/1650 objects degraded (10.909%), 23 pgs degraded 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: osd.5 (root=default,host=smithi193) is down 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 180/1650 objects degraded (10.909%), 23 pgs degraded 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-03-30T14:34:15.963 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.18 is active+undersized+degraded, acting [4,2] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-03-30T14:34:15.964 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-03-30T14:34:15.965 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-03-30T14:34:15.965 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-03-30T14:34:15.965 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-03-30T14:34:15.965 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-03-30T14:34:16.084 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:34:15.608+0000 7f3a5a61c7c0 -1 osd.5 128 log_to_monitors true 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: pgmap v244: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 475 op/s; 180/1650 objects degraded (10.909%) 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1335921513' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:16.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:16.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:16.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:16.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:16.182 INFO:teuthology.orchestra.run.smithi031.stdout: 5 263363 10.00 MB/sec execute 541 sec latency 52.734 ms 2024-03-30T14:34:16.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:16 smithi031 ceph-mon[67456]: pgmap v245: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 440 op/s; 178/1650 objects degraded (10.788%) 2024-03-30T14:34:16.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:16 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:16.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:16 smithi031 ceph-mon[67456]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-30T14:34:16.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:16 smithi031 ceph-mon[67456]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:16.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:16 smithi031 ceph-mon[67456]: from='osd.5 [v2:172.21.15.193:6808/3793136013,v1:172.21.15.193:6809/3793136013]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:34:16.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:16 smithi031 ceph-mon[67456]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:34:16.643 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:16 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1154323738' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:34:17.084 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:34:16.638+0000 7f3a4ba74700 -1 osd.5 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:34:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-mon[52977]: pgmap v245: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 440 op/s; 178/1650 objects degraded (10.788%) 2024-03-30T14:34:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-mon[52977]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-30T14:34:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-mon[52977]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-mon[52977]: from='osd.5 [v2:172.21.15.193:6808/3793136013,v1:172.21.15.193:6809/3793136013]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:34:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-mon[52977]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:34:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:16 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1154323738' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:34:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: 5 263673 10.00 MB/sec execute 542 sec latency 76.574 ms 2024-03-30T14:34:17.666 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:17 smithi031 ceph-mon[67456]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T14:34:17.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:17 smithi031 ceph-mon[67456]: osdmap e131: 8 total, 7 up, 8 in 2024-03-30T14:34:17.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:17 smithi031 ceph-mon[67456]: from='osd.5 [v2:172.21.15.193:6808/3793136013,v1:172.21.15.193:6809/3793136013]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:17.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:17 smithi031 ceph-mon[67456]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:17.667 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:17 smithi031 ceph-mon[67456]: failed to encode map e131 with expected crc 2024-03-30T14:34:18.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:17 smithi193 ceph-mon[52977]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T14:34:18.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:17 smithi193 ceph-mon[52977]: osdmap e131: 8 total, 7 up, 8 in 2024-03-30T14:34:18.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:17 smithi193 ceph-mon[52977]: from='osd.5 [v2:172.21.15.193:6808/3793136013,v1:172.21.15.193:6809/3793136013]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:18.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:17 smithi193 ceph-mon[52977]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:18.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:17 smithi193 ceph-mon[52977]: failed to encode map e131 with expected crc 2024-03-30T14:34:18.182 INFO:teuthology.orchestra.run.smithi031.stdout: 5 263900 9.99 MB/sec execute 543 sec latency 502.386 ms 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: failed to encode map e131 with expected crc 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: pgmap v247: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 482 op/s; 176/1620 objects degraded (10.864%) 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: failed to encode map e132 with expected crc 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: failed to encode map e132 with expected crc 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: failed to encode map e132 with expected crc 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: osd.5 [v2:172.21.15.193:6808/3793136013,v1:172.21.15.193:6809/3793136013] boot 2024-03-30T14:34:18.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:34:18.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:18 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:34:19.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: failed to encode map e131 with expected crc 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: pgmap v247: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 482 op/s; 176/1620 objects degraded (10.864%) 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: failed to encode map e132 with expected crc 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: failed to encode map e132 with expected crc 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: failed to encode map e132 with expected crc 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: osd.5 [v2:172.21.15.193:6808/3793136013,v1:172.21.15.193:6809/3793136013] boot 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:34:19.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:18 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:34:19.182 INFO:teuthology.orchestra.run.smithi031.stdout: 5 264050 9.98 MB/sec execute 544 sec latency 1039.240 ms 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: failed to encode map e133 with expected crc 2024-03-30T14:34:19.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 176/1620 objects degraded (10.864%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:19.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:19 smithi031 ceph-mon[67456]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:34:20.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: failed to encode map e133 with expected crc 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 176/1620 objects degraded (10.864%), 23 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:19 smithi193 ceph-mon[52977]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:34:20.182 INFO:teuthology.orchestra.run.smithi031.stdout: 5 264282 9.97 MB/sec execute 545 sec latency 752.442 ms 2024-03-30T14:34:20.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:20 smithi031 ceph-mon[67456]: pgmap v250: 97 pgs: 8 peering, 6 active+undersized, 17 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.9 MiB/s wr, 371 op/s; 132/1623 objects degraded (8.133%) 2024-03-30T14:34:20.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:20 smithi031 ceph-mon[67456]: failed to encode map e134 with expected crc 2024-03-30T14:34:20.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:20 smithi031 ceph-mon[67456]: failed to encode map e134 with expected crc 2024-03-30T14:34:20.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:20 smithi031 ceph-mon[67456]: failed to encode map e134 with expected crc 2024-03-30T14:34:21.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:20 smithi193 ceph-mon[52977]: pgmap v250: 97 pgs: 8 peering, 6 active+undersized, 17 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.9 MiB/s wr, 371 op/s; 132/1623 objects degraded (8.133%) 2024-03-30T14:34:21.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:20 smithi193 ceph-mon[52977]: failed to encode map e134 with expected crc 2024-03-30T14:34:21.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:20 smithi193 ceph-mon[52977]: failed to encode map e134 with expected crc 2024-03-30T14:34:21.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:20 smithi193 ceph-mon[52977]: failed to encode map e134 with expected crc 2024-03-30T14:34:21.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 264473 9.97 MB/sec execute 546 sec latency 540.804 ms 2024-03-30T14:34:21.901 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:21 smithi031 ceph-mon[67456]: failed to encode map e135 with expected crc 2024-03-30T14:34:21.902 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:21 smithi031 ceph-mon[67456]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:34:21.902 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:21 smithi031 ceph-mon[67456]: failed to encode map e135 with expected crc 2024-03-30T14:34:21.902 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:21 smithi031 ceph-mon[67456]: failed to encode map e135 with expected crc 2024-03-30T14:34:22.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:21 smithi193 ceph-mon[52977]: failed to encode map e135 with expected crc 2024-03-30T14:34:22.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:21 smithi193 ceph-mon[52977]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:34:22.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:21 smithi193 ceph-mon[52977]: failed to encode map e135 with expected crc 2024-03-30T14:34:22.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:21 smithi193 ceph-mon[52977]: failed to encode map e135 with expected crc 2024-03-30T14:34:22.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 264603 9.95 MB/sec execute 547 sec latency 1054.525 ms 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: pgmap v253: 97 pgs: 8 peering, 3 active+undersized, 1 active+undersized+remapped, 13 active+undersized+degraded, 72 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 484 op/s; 102/1623 objects degraded (6.285%); 223 KiB/s, 10 objects/s recovering 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: failed to encode map e136 with expected crc 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: failed to encode map e136 with expected crc 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: failed to encode map e136 with expected crc 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: failed to encode map e136 with expected crc 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: failed to encode map e136 with expected crc 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:34:22.663 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:22 smithi031 ceph-mon[67456]: failed to encode map e136 with expected crc 2024-03-30T14:34:23.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: pgmap v253: 97 pgs: 8 peering, 3 active+undersized, 1 active+undersized+remapped, 13 active+undersized+degraded, 72 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 484 op/s; 102/1623 objects degraded (6.285%); 223 KiB/s, 10 objects/s recovering 2024-03-30T14:34:23.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: failed to encode map e136 with expected crc 2024-03-30T14:34:23.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: failed to encode map e136 with expected crc 2024-03-30T14:34:23.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: failed to encode map e136 with expected crc 2024-03-30T14:34:23.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: failed to encode map e136 with expected crc 2024-03-30T14:34:23.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: failed to encode map e136 with expected crc 2024-03-30T14:34:23.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:34:23.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:22 smithi193 ceph-mon[52977]: failed to encode map e136 with expected crc 2024-03-30T14:34:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 264962 9.95 MB/sec execute 548 sec latency 37.812 ms 2024-03-30T14:34:23.696 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:23 smithi031 ceph-mon[67456]: pgmap v255: 97 pgs: 8 peering, 1 active+undersized+remapped, 88 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 454 B/s rd, 3.0 MiB/s wr, 301 op/s; 615 KiB/s, 34 objects/s recovering 2024-03-30T14:34:24.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:23 smithi193 ceph-mon[52977]: pgmap v255: 97 pgs: 8 peering, 1 active+undersized+remapped, 88 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 454 B/s rd, 3.0 MiB/s wr, 301 op/s; 615 KiB/s, 34 objects/s recovering 2024-03-30T14:34:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 265268 9.95 MB/sec execute 549 sec latency 50.179 ms 2024-03-30T14:34:24.721 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:24 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 102/1623 objects degraded (6.285%), 13 pgs degraded) 2024-03-30T14:34:25.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:24 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 102/1623 objects degraded (6.285%), 13 pgs degraded) 2024-03-30T14:34:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 265618 9.95 MB/sec execute 550 sec latency 43.518 ms 2024-03-30T14:34:25.747 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:25 smithi031 ceph-mon[67456]: pgmap v256: 97 pgs: 1 peering, 1 active+undersized+remapped, 95 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 420 op/s; 666 KiB/s, 32 objects/s recovering 2024-03-30T14:34:26.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:25 smithi193 ceph-mon[52977]: pgmap v256: 97 pgs: 1 peering, 1 active+undersized+remapped, 95 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 420 op/s; 666 KiB/s, 32 objects/s recovering 2024-03-30T14:34:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 265932 9.95 MB/sec execute 551 sec latency 55.901 ms 2024-03-30T14:34:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 266296 9.95 MB/sec execute 552 sec latency 56.142 ms 2024-03-30T14:34:28.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 266571 9.94 MB/sec execute 553 sec latency 51.506 ms 2024-03-30T14:34:28.707 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:28 smithi031 ceph-mon[67456]: pgmap v257: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.8 MiB/s wr, 350 op/s; 1.0 MiB/s, 23 objects/s recovering 2024-03-30T14:34:28.708 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:28.708 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:29.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:28 smithi193 ceph-mon[52977]: pgmap v257: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.8 MiB/s wr, 350 op/s; 1.0 MiB/s, 23 objects/s recovering 2024-03-30T14:34:29.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:29.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:29.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 266899 9.94 MB/sec execute 554 sec latency 71.054 ms 2024-03-30T14:34:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 267253 9.94 MB/sec execute 555 sec latency 56.913 ms 2024-03-30T14:34:30.818 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:30 smithi031 ceph-mon[67456]: pgmap v258: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.0 MiB/s wr, 376 op/s; 849 KiB/s, 18 objects/s recovering 2024-03-30T14:34:30.818 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:30 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:31.021 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:30 smithi193 ceph-mon[52977]: pgmap v258: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.0 MiB/s wr, 376 op/s; 849 KiB/s, 18 objects/s recovering 2024-03-30T14:34:31.021 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:30 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:31.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 267622 9.94 MB/sec execute 556 sec latency 53.790 ms 2024-03-30T14:34:31.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:31 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:31.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:31 smithi031 ceph-mon[67456]: Upgrade: osd.6 is safe to restart 2024-03-30T14:34:32.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:31 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-30T14:34:32.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:31 smithi193 ceph-mon[52977]: Upgrade: osd.6 is safe to restart 2024-03-30T14:34:32.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 267973 9.94 MB/sec execute 557 sec latency 42.138 ms 2024-03-30T14:34:32.615 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:32 smithi031 ceph-mon[67456]: pgmap v259: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 422 op/s; 720 KiB/s, 15 objects/s recovering 2024-03-30T14:34:32.615 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:32.615 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:34:32.615 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:34:33.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:32 smithi193 ceph-mon[52977]: pgmap v259: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 422 op/s; 720 KiB/s, 15 objects/s recovering 2024-03-30T14:34:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:34:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:34:33.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 268292 9.94 MB/sec execute 558 sec latency 66.018 ms 2024-03-30T14:34:33.624 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:33 smithi031 ceph-mon[67456]: Upgrade: Updating osd.6 2024-03-30T14:34:33.625 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:33 smithi031 ceph-mon[67456]: Deploying daemon osd.6 on smithi193 2024-03-30T14:34:33.625 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:33 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:33.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:33 smithi193 ceph-mon[52977]: Upgrade: Updating osd.6 2024-03-30T14:34:33.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:33 smithi193 ceph-mon[52977]: Deploying daemon osd.6 on smithi193 2024-03-30T14:34:33.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:33 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:34.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 268605 9.93 MB/sec execute 559 sec latency 46.272 ms 2024-03-30T14:34:34.640 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:34 smithi031 ceph-mon[67456]: pgmap v260: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 432 op/s; 487 KiB/s, 6 objects/s recovering 2024-03-30T14:34:34.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:34 smithi193 ceph-mon[52977]: pgmap v260: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 432 op/s; 487 KiB/s, 6 objects/s recovering 2024-03-30T14:34:35.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 268909 9.93 MB/sec execute 560 sec latency 42.196 ms 2024-03-30T14:34:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 269341 9.94 MB/sec execute 561 sec latency 50.324 ms 2024-03-30T14:34:36.585 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:36 smithi193 systemd[1]: Stopping Ceph osd.6 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:36.673 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:36 smithi031 ceph-mon[67456]: pgmap v261: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 483 op/s; 444 KiB/s, 3 objects/s recovering 2024-03-30T14:34:36.673 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:36 smithi031 ceph-mon[67456]: osd.6 marked itself down and dead 2024-03-30T14:34:37.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:36 smithi193 ceph-mon[52977]: pgmap v261: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 483 op/s; 444 KiB/s, 3 objects/s recovering 2024-03-30T14:34:37.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:36 smithi193 ceph-mon[52977]: osd.6 marked itself down and dead 2024-03-30T14:34:37.085 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[37122]: 2024-03-30T14:34:36.587+0000 7fc72dbf7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:34:37.085 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[37122]: 2024-03-30T14:34:36.587+0000 7fc72dbf7700 -1 osd.6 136 *** Got signal Terminated *** 2024-03-30T14:34:37.085 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[37122]: 2024-03-30T14:34:36.587+0000 7fc72dbf7700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:34:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 269608 9.93 MB/sec execute 562 sec latency 550.290 ms 2024-03-30T14:34:37.534 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:37 smithi193 podman[62014]: 2024-03-30 14:34:37.270536488 +0000 UTC m=+0.825463509 container died 87eed05f7b3586abd70e2df8089bb35348a6fade1040ee7dee952dfdb2af830f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, version=8, io.openshift.expose-services=, name=centos-stream, vcs-type=git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64) 2024-03-30T14:34:37.834 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:37 smithi193 podman[62014]: 2024-03-30 14:34:37.61963785 +0000 UTC m=+1.174564864 container cleanup 87eed05f7b3586abd70e2df8089bb35348a6fade1040ee7dee952dfdb2af830f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.component=centos-stream-container, name=centos-stream, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-30T14:34:37.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:37 smithi193 bash[62014]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6 2024-03-30T14:34:37.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:37 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:34:37.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:37 smithi193 ceph-mon[52977]: failed to encode map e137 with expected crc 2024-03-30T14:34:37.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:37 smithi193 ceph-mon[52977]: osdmap e137: 8 total, 7 up, 8 in 2024-03-30T14:34:37.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:37 smithi193 ceph-mon[52977]: failed to encode map e137 with expected crc 2024-03-30T14:34:37.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:37 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:34:37.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:37 smithi031 ceph-mon[67456]: failed to encode map e137 with expected crc 2024-03-30T14:34:37.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:37 smithi031 ceph-mon[67456]: osdmap e137: 8 total, 7 up, 8 in 2024-03-30T14:34:37.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:37 smithi031 ceph-mon[67456]: failed to encode map e137 with expected crc 2024-03-30T14:34:38.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 269872 9.93 MB/sec execute 563 sec latency 1014.330 ms 2024-03-30T14:34:38.334 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:37 smithi193 podman[62034]: 2024-03-30 14:34:37.938249416 +0000 UTC m=+0.666288739 container remove 87eed05f7b3586abd70e2df8089bb35348a6fade1040ee7dee952dfdb2af830f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, distribution-scope=public, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-03-30T14:34:38.644 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:38 smithi193 podman[62120]: 2024-03-30 14:34:38.252555949 +0000 UTC m=+0.021801346 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:38.644 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:38 smithi193 podman[62120]: 2024-03-30 14:34:38.389478057 +0000 UTC m=+0.158723459 container create de01c7e3ec106cde8ae35294780a835e095c6a0043fe778ca6501cd5341365cb (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:38.645 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:38 smithi031 ceph-mon[67456]: pgmap v263: 97 pgs: 8 peering, 7 stale+active+clean, 82 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.0 MiB/s wr, 480 op/s 2024-03-30T14:34:38.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:38 smithi031 ceph-mon[67456]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:38 smithi031 ceph-mon[67456]: osdmap e138: 8 total, 7 up, 8 in 2024-03-30T14:34:38.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:38 smithi031 ceph-mon[67456]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:38 smithi031 ceph-mon[67456]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:38 smithi031 ceph-mon[67456]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:38 smithi031 ceph-mon[67456]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.898 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:38 smithi193 ceph-mon[52977]: pgmap v263: 97 pgs: 8 peering, 7 stale+active+clean, 82 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.0 MiB/s wr, 480 op/s 2024-03-30T14:34:38.898 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:38 smithi193 ceph-mon[52977]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.899 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:38 smithi193 ceph-mon[52977]: osdmap e138: 8 total, 7 up, 8 in 2024-03-30T14:34:38.899 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:38 smithi193 ceph-mon[52977]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.899 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:38 smithi193 ceph-mon[52977]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.899 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:38 smithi193 ceph-mon[52977]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.899 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:38 smithi193 ceph-mon[52977]: failed to encode map e138 with expected crc 2024-03-30T14:34:38.899 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:38 smithi193 podman[62120]: 2024-03-30 14:34:38.809967014 +0000 UTC m=+0.579212409 container init de01c7e3ec106cde8ae35294780a835e095c6a0043fe778ca6501cd5341365cb (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, ceph=True) 2024-03-30T14:34:38.899 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:38 smithi193 podman[62120]: 2024-03-30 14:34:38.813970603 +0000 UTC m=+0.583216001 container start de01c7e3ec106cde8ae35294780a835e095c6a0043fe778ca6501cd5341365cb (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-30T14:34:38.899 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:38 smithi193 podman[62120]: 2024-03-30 14:34:38.90154309 +0000 UTC m=+0.670788491 container attach de01c7e3ec106cde8ae35294780a835e095c6a0043fe778ca6501cd5341365cb (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-4e50bbc) 2024-03-30T14:34:39.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 270228 9.92 MB/sec execute 564 sec latency 29.775 ms 2024-03-30T14:34:39.334 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:39 smithi193 podman[62120]: 2024-03-30 14:34:39.147581907 +0000 UTC m=+0.916827303 container died de01c7e3ec106cde8ae35294780a835e095c6a0043fe778ca6501cd5341365cb (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, ceph=True) 2024-03-30T14:34:39.915 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:39 smithi031 ceph-mon[67456]: pgmap v265: 97 pgs: 5 active+undersized, 11 active+undersized+degraded, 8 peering, 2 stale+active+clean, 71 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.2 MiB/s wr, 506 op/s; 81/1671 objects degraded (4.847%) 2024-03-30T14:34:39.915 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:39 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 81/1671 objects degraded (4.847%), 11 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:40.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:39 smithi193 ceph-mon[52977]: pgmap v265: 97 pgs: 5 active+undersized, 11 active+undersized+degraded, 8 peering, 2 stale+active+clean, 71 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.2 MiB/s wr, 506 op/s; 81/1671 objects degraded (4.847%) 2024-03-30T14:34:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:39 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 81/1671 objects degraded (4.847%), 11 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:40.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 270607 9.92 MB/sec execute 565 sec latency 31.440 ms 2024-03-30T14:34:40.353 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 podman[62145]: 2024-03-30 14:34:40.104871754 +0000 UTC m=+0.952143768 container remove de01c7e3ec106cde8ae35294780a835e095c6a0043fe778ca6501cd5341365cb (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True) 2024-03-30T14:34:40.354 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.6.service: Deactivated successfully. 2024-03-30T14:34:40.354 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 systemd[1]: Stopped Ceph osd.6 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:40.354 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.6.service: Consumed 5min 4.060s CPU time. 2024-03-30T14:34:40.354 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 systemd[1]: Starting Ceph osd.6 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:40.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 podman[62241]: 2024-03-30 14:34:40.584526114 +0000 UTC m=+0.118006058 container create 692c7f98fd652aeea5d983830c2d6b399cd32984ae349acd8df6cae7917ecd94 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326) 2024-03-30T14:34:40.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 podman[62241]: 2024-03-30 14:34:40.487889898 +0000 UTC m=+0.021369856 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:41.183 INFO:teuthology.orchestra.run.smithi031.stdout: 5 271047 9.93 MB/sec execute 566 sec latency 36.836 ms 2024-03-30T14:34:41.335 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 podman[62241]: 2024-03-30 14:34:40.845536864 +0000 UTC m=+0.379016820 container init 692c7f98fd652aeea5d983830c2d6b399cd32984ae349acd8df6cae7917ecd94 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:41.335 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 podman[62241]: 2024-03-30 14:34:40.850056744 +0000 UTC m=+0.383536688 container start 692c7f98fd652aeea5d983830c2d6b399cd32984ae349acd8df6cae7917ecd94 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:34:41.335 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:40 smithi193 podman[62241]: 2024-03-30 14:34:40.937418912 +0000 UTC m=+0.470898872 container attach 692c7f98fd652aeea5d983830c2d6b399cd32984ae349acd8df6cae7917ecd94 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc) 2024-03-30T14:34:41.705 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate[62255]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate[62255]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 bash[62241]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 bash[62241]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 bash[62241]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 bash[62241]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 bash[62241]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 bash[62241]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 bash[62241]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate[62255]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate[62255]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate[62255]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate[62255]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-30T14:34:41.706 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate[62255]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-30T14:34:41.707 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:41 smithi193 podman[62241]: 2024-03-30 14:34:41.377748003 +0000 UTC m=+0.911227946 container died 692c7f98fd652aeea5d983830c2d6b399cd32984ae349acd8df6cae7917ecd94 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326) 2024-03-30T14:34:42.184 INFO:teuthology.orchestra.run.smithi031.stdout: 5 271366 9.93 MB/sec execute 567 sec latency 105.950 ms 2024-03-30T14:34:42.215 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:42 smithi031 ceph-mon[67456]: pgmap v266: 97 pgs: 8 active+undersized, 19 active+undersized+degraded, 8 peering, 62 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 534 op/s; 141/1695 objects degraded (8.319%) 2024-03-30T14:34:42.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:42 smithi193 ceph-mon[52977]: pgmap v266: 97 pgs: 8 active+undersized, 19 active+undersized+degraded, 8 peering, 62 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 534 op/s; 141/1695 objects degraded (8.319%) 2024-03-30T14:34:42.308 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 podman[62387]: 2024-03-30 14:34:42.00159482 +0000 UTC m=+0.618967188 container remove 692c7f98fd652aeea5d983830c2d6b399cd32984ae349acd8df6cae7917ecd94 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:34:42.308 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 podman[62485]: 2024-03-30 14:34:42.119968015 +0000 UTC m=+0.017772407 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:42.308 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 podman[62485]: 2024-03-30 14:34:42.311102424 +0000 UTC m=+0.208906789 container create 8f73251ee641f51400b7b0cf1e715c583663793379b444f0b0ccc62f8347c41c (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:34:42.834 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 podman[62485]: 2024-03-30 14:34:42.56571729 +0000 UTC m=+0.463521655 container init 8f73251ee641f51400b7b0cf1e715c583663793379b444f0b0ccc62f8347c41c (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-03-30T14:34:42.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 podman[62485]: 2024-03-30 14:34:42.570502859 +0000 UTC m=+0.468307224 container start 8f73251ee641f51400b7b0cf1e715c583663793379b444f0b0ccc62f8347c41c (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326) 2024-03-30T14:34:42.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 ceph-osd[62504]: -- 172.21.15.193:0/3640803595 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557bf724f680 con 0x557bf7218c00 2024-03-30T14:34:42.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 bash[62485]: 8f73251ee641f51400b7b0cf1e715c583663793379b444f0b0ccc62f8347c41c 2024-03-30T14:34:42.835 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:42 smithi193 systemd[1]: Started Ceph osd.6 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:43.184 INFO:teuthology.orchestra.run.smithi031.stdout: 5 271770 9.93 MB/sec execute 568 sec latency 52.021 ms 2024-03-30T14:34:43.584 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:43 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:34:43.128+0000 7fb270f577c0 -1 Falling back to public interface 2024-03-30T14:34:43.726 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:43.726 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:43.726 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:43.726 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:43.726 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:43.924 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:43.924 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:43.924 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:43.924 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:43.924 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: 5 272134 9.92 MB/sec execute 569 sec latency 45.011 ms 2024-03-30T14:34:44.736 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:44 smithi031 ceph-mon[67456]: pgmap v267: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 513 op/s; 196/1668 objects degraded (11.751%) 2024-03-30T14:34:44.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:44 smithi193 ceph-mon[52977]: pgmap v267: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 513 op/s; 196/1668 objects degraded (11.751%) 2024-03-30T14:34:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: 5 272465 9.92 MB/sec execute 570 sec latency 48.985 ms 2024-03-30T14:34:45.760 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:45 smithi031 ceph-mon[67456]: pgmap v268: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 465 op/s; 197/1668 objects degraded (11.811%) 2024-03-30T14:34:45.760 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:45 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 197/1668 objects degraded (11.811%), 25 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:45.919 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:45 smithi193 ceph-mon[52977]: pgmap v268: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 465 op/s; 197/1668 objects degraded (11.811%) 2024-03-30T14:34:45.919 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:45 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 197/1668 objects degraded (11.811%), 25 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: 5 272777 9.92 MB/sec execute 571 sec latency 42.109 ms 2024-03-30T14:34:46.614 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:34:46.782 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:46 smithi031 ceph-mon[67456]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:46.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:46 smithi193 ceph-mon[52977]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:47.185 INFO:teuthology.orchestra.run.smithi031.stdout: 5 273074 9.92 MB/sec execute 572 sec latency 50.988 ms 2024-03-30T14:34:47.714 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:34:47.714 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (8m) 112s ago 18m 23.7M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:34:47.714 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (5m) 112s ago 17m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (4m) 0s ago 17m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 112s ago 17m - - 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (12m) 112s ago 12m 40.4M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (12m) 0s ago 12m 15.9M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (7m) 112s ago 19m 507M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (6m) 0s ago 17m 452M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (6m) 112s ago 19m 52.1M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (6m) 0s ago 17m 44.5M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (11m) 112s ago 11m 280M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (11m) 0s ago 11m 85.1M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (8m) 112s ago 17m 17.3M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (8m) 0s ago 17m 17.1M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (4m) 112s ago 16m 363M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (3m) 112s ago 16m 303M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:34:47.715 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (2m) 112s ago 15m 227M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:34:47.716 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (2m) 112s ago 15m 12.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:34:47.716 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (79s) 0s ago 14m 295M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:34:47.716 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (42s) 0s ago 14m 265M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:34:47.716 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (5s) 0s ago 14m 12.4M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:34:47.716 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (13m) 0s ago 13m 598M 2704M 17.2.0 e1d6a67b021e 1d6e1da533f8 2024-03-30T14:34:47.716 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (7m) 112s ago 16m 70.0M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:34:48.186 INFO:teuthology.orchestra.run.smithi031.stdout: 5 273384 9.92 MB/sec execute 573 sec latency 58.072 ms 2024-03-30T14:34:48.306 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:48 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:48.306 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:48 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:48.306 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:48 smithi031 ceph-mon[67456]: pgmap v269: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 538 op/s; 199/1635 objects degraded (12.171%) 2024-03-30T14:34:48.306 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:48 smithi031 ceph-mon[67456]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 6 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 10 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:34:48.331 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:34:48.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:48 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:48.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:48 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:48.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:48 smithi193 ceph-mon[52977]: pgmap v269: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 538 op/s; 199/1635 objects degraded (12.171%) 2024-03-30T14:34:48.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:48 smithi193 ceph-mon[52977]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "13/23 daemons upgraded", 2024-03-30T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:34:48.823 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:34:48.823 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:34:49.188 INFO:teuthology.orchestra.run.smithi031.stdout: 5 273663 9.91 MB/sec execute 574 sec latency 46.448 ms 2024-03-30T14:34:49.331 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:49 smithi031 ceph-mon[67456]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:49.331 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:49.331 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:49.331 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:49 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1124281704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 202/1686 objects degraded (11.981%), 25 pgs degraded 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: osd.6 (root=default,host=smithi193) is down 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 202/1686 objects degraded (11.981%), 25 pgs degraded 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-03-30T14:34:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-03-30T14:34:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-03-30T14:34:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-03-30T14:34:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-03-30T14:34:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-03-30T14:34:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.a is active+undersized+degraded, acting [1,7] 2024-03-30T14:34:49.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:49 smithi193 ceph-mon[52977]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:49.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:49.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:49.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:49 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1124281704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:50.190 INFO:teuthology.orchestra.run.smithi031.stdout: 5 274030 9.91 MB/sec execute 575 sec latency 38.672 ms 2024-03-30T14:34:50.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:50 smithi193 ceph-mon[52977]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:50.350 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:50 smithi031 ceph-mon[67456]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:34:50.350 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:50 smithi031 ceph-mon[67456]: pgmap v270: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 490 op/s; 202/1686 objects degraded (11.981%) 2024-03-30T14:34:50.350 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:50 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/548413960' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:34:50.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:50 smithi193 ceph-mon[52977]: pgmap v270: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 490 op/s; 202/1686 objects degraded (11.981%) 2024-03-30T14:34:50.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:50 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/548413960' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:34:50.584 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:50 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:34:50.549+0000 7fb270f577c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-30T14:34:51.197 INFO:teuthology.orchestra.run.smithi031.stdout: 5 274360 9.90 MB/sec execute 576 sec latency 60.924 ms 2024-03-30T14:34:51.392 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:51 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 198/1668 objects degraded (11.871%), 25 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:51.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:51 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 198/1668 objects degraded (11.871%), 25 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:52.160 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:51 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:34:51.987+0000 7fb270f577c0 -1 osd.6 136 log_to_monitors true 2024-03-30T14:34:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: 5 274734 9.91 MB/sec execute 577 sec latency 50.359 ms 2024-03-30T14:34:52.411 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: pgmap v271: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 535 op/s; 198/1668 objects degraded (11.871%) 2024-03-30T14:34:52.411 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='osd.6 [v2:172.21.15.193:6816/342680429,v1:172.21.15.193:6817/342680429]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:34:52.411 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:34:52.411 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:52.411 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:52.412 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:34:52.412 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:34:52.412 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:52.412 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:52.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: pgmap v271: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 535 op/s; 198/1668 objects degraded (11.871%) 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='osd.6 [v2:172.21.15.193:6816/342680429,v1:172.21.15.193:6817/342680429]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:52.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:53.197 INFO:teuthology.orchestra.run.smithi031.stdout: 5 274981 9.90 MB/sec execute 578 sec latency 150.552 ms 2024-03-30T14:34:53.335 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:34:52 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:34:52.855+0000 7fb25e3a7700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:34:53.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: pgmap v272: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 516 op/s; 194/1653 objects degraded (11.736%) 2024-03-30T14:34:53.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.2 MiB/s wr, 538 op/s; 194/1653 objects degraded (11.736%) 2024-03-30T14:34:53.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T14:34:53.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='osd.6 [v2:172.21.15.193:6816/342680429,v1:172.21.15.193:6817/342680429]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: osdmap e139: 8 total, 7 up, 8 in 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: failed to encode map e139 with expected crc 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-30T14:34:53.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:53 smithi031 ceph-mon[67456]: from='osd.6 ' entity='osd.6' 2024-03-30T14:34:53.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: pgmap v272: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 516 op/s; 194/1653 objects degraded (11.736%) 2024-03-30T14:34:53.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.2 MiB/s wr, 538 op/s; 194/1653 objects degraded (11.736%) 2024-03-30T14:34:53.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T14:34:53.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='osd.6 [v2:172.21.15.193:6816/342680429,v1:172.21.15.193:6817/342680429]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:53.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: osdmap e139: 8 total, 7 up, 8 in 2024-03-30T14:34:53.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:34:53.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: failed to encode map e139 with expected crc 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-30T14:34:53.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:53 smithi193 ceph-mon[52977]: from='osd.6 ' entity='osd.6' 2024-03-30T14:34:54.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 275314 9.91 MB/sec execute 579 sec latency 371.726 ms 2024-03-30T14:34:54.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:34:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: failed to encode map e140 with expected crc 2024-03-30T14:34:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: osd.6 [v2:172.21.15.193:6816/342680429,v1:172.21.15.193:6817/342680429] boot 2024-03-30T14:34:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:34:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:34:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: failed to encode map e140 with expected crc 2024-03-30T14:34:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: failed to encode map e140 with expected crc 2024-03-30T14:34:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:54 smithi193 ceph-mon[52977]: failed to encode map e140 with expected crc 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: failed to encode map e140 with expected crc 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: osd.6 [v2:172.21.15.193:6816/342680429,v1:172.21.15.193:6817/342680429] boot 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: failed to encode map e140 with expected crc 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: failed to encode map e140 with expected crc 2024-03-30T14:34:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:54 smithi031 ceph-mon[67456]: failed to encode map e140 with expected crc 2024-03-30T14:34:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 275314 9.89 MB/sec execute 580 sec latency 1371.775 ms 2024-03-30T14:34:55.814 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:55 smithi031 ceph-mon[67456]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 619 op/s; 194/1644 objects degraded (11.800%) 2024-03-30T14:34:55.814 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:55 smithi031 ceph-mon[67456]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:34:55.814 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:55 smithi031 ceph-mon[67456]: failed to encode map e141 with expected crc 2024-03-30T14:34:55.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:55 smithi193 ceph-mon[52977]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 619 op/s; 194/1644 objects degraded (11.800%) 2024-03-30T14:34:55.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:55 smithi193 ceph-mon[52977]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:34:55.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:55 smithi193 ceph-mon[52977]: failed to encode map e141 with expected crc 2024-03-30T14:34:56.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 275476 9.88 MB/sec execute 581 sec latency 1847.491 ms 2024-03-30T14:34:56.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:56 smithi031 ceph-mon[67456]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:34:56.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:56 smithi031 ceph-mon[67456]: failed to encode map e142 with expected crc 2024-03-30T14:34:56.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:56 smithi031 ceph-mon[67456]: failed to encode map e142 with expected crc 2024-03-30T14:34:56.581 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:56 smithi031 ceph-mon[67456]: failed to encode map e142 with expected crc 2024-03-30T14:34:56.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:56 smithi193 ceph-mon[52977]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:34:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:56 smithi193 ceph-mon[52977]: failed to encode map e142 with expected crc 2024-03-30T14:34:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:56 smithi193 ceph-mon[52977]: failed to encode map e142 with expected crc 2024-03-30T14:34:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:56 smithi193 ceph-mon[52977]: failed to encode map e142 with expected crc 2024-03-30T14:34:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 275758 9.87 MB/sec execute 582 sec latency 475.983 ms 2024-03-30T14:34:57.813 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:57 smithi031 ceph-mon[67456]: pgmap v279: 97 pgs: 2 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 remapped+peering, 1 active+recovery_wait+degraded, 7 active+undersized, 19 active+undersized+degraded, 65 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s wr, 365 op/s; 169/1641 objects degraded (10.299%); 41 KiB/s, 5 objects/s recovering 2024-03-30T14:34:57.813 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:57 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 169/1641 objects degraded (10.299%), 21 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:57.813 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:57 smithi031 ceph-mon[67456]: failed to encode map e143 with expected crc 2024-03-30T14:34:57.813 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:57 smithi031 ceph-mon[67456]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:34:57.813 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:57 smithi031 ceph-mon[67456]: failed to encode map e143 with expected crc 2024-03-30T14:34:57.813 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:57 smithi031 ceph-mon[67456]: failed to encode map e143 with expected crc 2024-03-30T14:34:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:57 smithi193 ceph-mon[52977]: pgmap v279: 97 pgs: 2 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 remapped+peering, 1 active+recovery_wait+degraded, 7 active+undersized, 19 active+undersized+degraded, 65 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s wr, 365 op/s; 169/1641 objects degraded (10.299%); 41 KiB/s, 5 objects/s recovering 2024-03-30T14:34:57.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:57 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 169/1641 objects degraded (10.299%), 21 pgs degraded (PG_DEGRADED) 2024-03-30T14:34:57.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:57 smithi193 ceph-mon[52977]: failed to encode map e143 with expected crc 2024-03-30T14:34:57.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:57 smithi193 ceph-mon[52977]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:34:57.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:57 smithi193 ceph-mon[52977]: failed to encode map e143 with expected crc 2024-03-30T14:34:57.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:57 smithi193 ceph-mon[52977]: failed to encode map e143 with expected crc 2024-03-30T14:34:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 275995 9.87 MB/sec execute 583 sec latency 878.427 ms 2024-03-30T14:34:58.590 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:58 smithi031 ceph-mon[67456]: failed to encode map e144 with expected crc 2024-03-30T14:34:58.590 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:58 smithi031 ceph-mon[67456]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:34:58.590 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:58 smithi031 ceph-mon[67456]: failed to encode map e144 with expected crc 2024-03-30T14:34:58.590 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:58 smithi031 ceph-mon[67456]: failed to encode map e144 with expected crc 2024-03-30T14:34:58.590 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:58.590 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:58.834 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:58 smithi193 ceph-mon[52977]: failed to encode map e144 with expected crc 2024-03-30T14:34:58.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:58 smithi193 ceph-mon[52977]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:34:58.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:58 smithi193 ceph-mon[52977]: failed to encode map e144 with expected crc 2024-03-30T14:34:58.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:58 smithi193 ceph-mon[52977]: failed to encode map e144 with expected crc 2024-03-30T14:34:58.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:34:58.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:34:59.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 276338 9.87 MB/sec execute 584 sec latency 70.557 ms 2024-03-30T14:34:59.852 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:34:59 smithi031 ceph-mon[67456]: pgmap v282: 97 pgs: 3 peering, 2 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 remapped+peering, 1 active+recovery_wait+degraded, 4 active+undersized+degraded, 84 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.9 MiB/s wr, 210 op/s; 64/1635 objects degraded (3.914%); 337 KiB/s, 32 objects/s recovering 2024-03-30T14:35:00.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:34:59 smithi193 ceph-mon[52977]: pgmap v282: 97 pgs: 3 peering, 2 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 remapped+peering, 1 active+recovery_wait+degraded, 4 active+undersized+degraded, 84 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.9 MiB/s wr, 210 op/s; 64/1635 objects degraded (3.914%); 337 KiB/s, 32 objects/s recovering 2024-03-30T14:35:00.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 276705 9.87 MB/sec execute 585 sec latency 54.406 ms 2024-03-30T14:35:01.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 277060 9.87 MB/sec execute 586 sec latency 34.908 ms 2024-03-30T14:35:01.634 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:01 smithi031 ceph-mon[67456]: pgmap v283: 97 pgs: 3 peering, 2 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 91 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.9 MiB/s wr, 237 op/s; 30/1623 objects degraded (1.848%); 905 KiB/s, 41 keys/s, 34 objects/s recovering 2024-03-30T14:35:02.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:01 smithi193 ceph-mon[52977]: pgmap v283: 97 pgs: 3 peering, 2 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 91 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.9 MiB/s wr, 237 op/s; 30/1623 objects degraded (1.848%); 905 KiB/s, 41 keys/s, 34 objects/s recovering 2024-03-30T14:35:02.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 277449 9.87 MB/sec execute 587 sec latency 40.849 ms 2024-03-30T14:35:02.680 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:02 smithi031 ceph-mon[67456]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 347 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.2 MiB/s wr, 393 op/s; 3.2 MiB/s, 35 keys/s, 33 objects/s recovering 2024-03-30T14:35:02.680 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:02 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/1623 objects degraded (1.848%), 1 pg degraded) 2024-03-30T14:35:03.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:02 smithi193 ceph-mon[52977]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 347 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.2 MiB/s wr, 393 op/s; 3.2 MiB/s, 35 keys/s, 33 objects/s recovering 2024-03-30T14:35:03.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:02 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/1623 objects degraded (1.848%), 1 pg degraded) 2024-03-30T14:35:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 277826 9.87 MB/sec execute 588 sec latency 56.036 ms 2024-03-30T14:35:04.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 278162 9.86 MB/sec execute 589 sec latency 57.123 ms 2024-03-30T14:35:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: 5 278465 9.86 MB/sec execute 590 sec latency 48.992 ms 2024-03-30T14:35:05.502 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:05 smithi031 ceph-mon[67456]: pgmap v285: 97 pgs: 97 active+clean; 347 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 450 op/s; 2.7 MiB/s, 29 keys/s, 28 objects/s recovering 2024-03-30T14:35:05.603 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:05 smithi193 ceph-mon[52977]: pgmap v285: 97 pgs: 97 active+clean; 347 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 450 op/s; 2.7 MiB/s, 29 keys/s, 28 objects/s recovering 2024-03-30T14:35:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 278809 9.86 MB/sec execute 591 sec latency 49.487 ms 2024-03-30T14:35:07.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 279110 9.86 MB/sec execute 592 sec latency 43.995 ms 2024-03-30T14:35:07.550 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:07 smithi031 ceph-mon[67456]: pgmap v286: 97 pgs: 97 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.5 MiB/s wr, 451 op/s; 2.2 MiB/s, 24 keys/s, 23 objects/s recovering 2024-03-30T14:35:07.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:07 smithi193 ceph-mon[52977]: pgmap v286: 97 pgs: 97 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.5 MiB/s wr, 451 op/s; 2.2 MiB/s, 24 keys/s, 23 objects/s recovering 2024-03-30T14:35:08.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 279463 9.86 MB/sec execute 593 sec latency 50.144 ms 2024-03-30T14:35:08.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:08 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:35:08.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:08 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:35:08.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:08 smithi193 ceph-mon[52977]: Upgrade: osd.7 is safe to restart 2024-03-30T14:35:08.567 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:08 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:35:08.568 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:08 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-30T14:35:08.568 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:08 smithi031 ceph-mon[67456]: Upgrade: osd.7 is safe to restart 2024-03-30T14:35:09.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 279721 9.86 MB/sec execute 594 sec latency 45.314 ms 2024-03-30T14:35:09.335 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:09 smithi031 ceph-mon[67456]: pgmap v287: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 447 op/s; 1.9 MiB/s, 21 keys/s, 14 objects/s recovering 2024-03-30T14:35:09.335 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:09 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:09.335 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:09 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:35:09.335 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:09 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:09.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:09 smithi193 ceph-mon[52977]: pgmap v287: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 447 op/s; 1.9 MiB/s, 21 keys/s, 14 objects/s recovering 2024-03-30T14:35:09.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:09 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:09.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:09 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:35:09.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:09 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:10.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 280027 9.86 MB/sec execute 595 sec latency 62.546 ms 2024-03-30T14:35:10.374 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:10 smithi031 ceph-mon[67456]: Upgrade: Updating osd.7 2024-03-30T14:35:10.374 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:10 smithi031 ceph-mon[67456]: Deploying daemon osd.7 on smithi193 2024-03-30T14:35:10.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:10 smithi193 ceph-mon[52977]: Upgrade: Updating osd.7 2024-03-30T14:35:10.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:10 smithi193 ceph-mon[52977]: Deploying daemon osd.7 on smithi193 2024-03-30T14:35:11.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 280327 9.85 MB/sec execute 596 sec latency 43.492 ms 2024-03-30T14:35:11.402 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:11 smithi031 ceph-mon[67456]: pgmap v288: 97 pgs: 97 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.6 MiB/s wr, 464 op/s; 1.7 MiB/s, 19 keys/s, 9 objects/s recovering 2024-03-30T14:35:11.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:11 smithi193 ceph-mon[52977]: pgmap v288: 97 pgs: 97 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.6 MiB/s wr, 464 op/s; 1.7 MiB/s, 19 keys/s, 9 objects/s recovering 2024-03-30T14:35:12.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 280566 9.85 MB/sec execute 597 sec latency 114.608 ms 2024-03-30T14:35:13.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 280839 9.84 MB/sec execute 598 sec latency 69.768 ms 2024-03-30T14:35:13.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:13 smithi193 systemd[1]: Stopping Ceph osd.7 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:35:13.696 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:13 smithi031 ceph-mon[67456]: pgmap v289: 97 pgs: 97 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.3 MiB/s wr, 534 op/s; 1.4 MiB/s, 4 objects/s recovering 2024-03-30T14:35:13.696 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:13.696 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:13.696 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:13 smithi193 ceph-mon[52977]: pgmap v289: 97 pgs: 97 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.3 MiB/s wr, 534 op/s; 1.4 MiB/s, 4 objects/s recovering 2024-03-30T14:35:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:14.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:14.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 281130 9.83 MB/sec execute 599 sec latency 47.573 ms 2024-03-30T14:35:14.334 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:14 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[40205]: 2024-03-30T14:35:14.079+0000 7f5b16149700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:35:14.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:14 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[40205]: 2024-03-30T14:35:14.079+0000 7f5b16149700 -1 osd.7 144 *** Got signal Terminated *** 2024-03-30T14:35:14.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:14 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[40205]: 2024-03-30T14:35:14.079+0000 7f5b16149700 -1 osd.7 144 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:35:14.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:14 smithi031 ceph-mon[67456]: osd.7 marked itself down and dead 2024-03-30T14:35:15.084 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:14 smithi193 ceph-mon[52977]: osd.7 marked itself down and dead 2024-03-30T14:35:15.200 INFO:teuthology.orchestra.run.smithi031.stdout: 5 cleanup 600 sec 2024-03-30T14:35:15.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:15 smithi193 podman[64290]: 2024-03-30 14:35:15.307773352 +0000 UTC m=+1.932224560 container died 1d6e1da533f85104a5768c4af80339dc65a7110ec3abfac5ea44c2bbc316343f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux ) 2024-03-30T14:35:15.859 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:15 smithi031 ceph-mon[67456]: pgmap v290: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 492 op/s; 8.3 KiB/s, 1 objects/s recovering 2024-03-30T14:35:15.859 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:15 smithi031 ceph-mon[67456]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:35:15.859 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:15 smithi031 ceph-mon[67456]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-30T14:35:15.859 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:15 smithi031 ceph-mon[67456]: osdmap e145: 8 total, 7 up, 8 in 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: 0 cleanup 601 sec 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: Operation Count AvgLat MaxLat 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: ---------------------------------------- 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: NTCreateX 188907 4.677 2228.713 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: Close 138674 0.146 21.973 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: Rename 8012 9.996 1054.514 2024-03-30T14:35:15.953 INFO:teuthology.orchestra.run.smithi031.stdout: Unlink 38225 7.054 1045.884 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: Qpathinfo 171306 1.198 2129.811 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: Qfileinfo 29866 0.008 1.059 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: Qfsinfo 31461 0.850 1451.747 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: Sfileinfo 15383 12.353 1048.596 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: Find 66275 1.044 21.372 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: WriteX 93372 13.393 2344.137 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: ReadX 296710 0.010 5.173 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: LockX 616 0.416 5.324 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: UnlockX 616 0.347 4.393 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: Flush 13247 0.002 0.587 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:35:15.954 INFO:teuthology.orchestra.run.smithi031.stdout:Throughput 9.83209 MB/sec 5 clients 5 procs max_latency=2344.145 ms 2024-03-30T14:35:15.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-03-30T14:35:15.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-03-30T14:35:16.018 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:15 smithi193 ceph-mon[52977]: pgmap v290: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 492 op/s; 8.3 KiB/s, 1 objects/s recovering 2024-03-30T14:35:16.018 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:15 smithi193 ceph-mon[52977]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T14:35:16.018 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:15 smithi193 ceph-mon[52977]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-30T14:35:16.018 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:15 smithi193 ceph-mon[52977]: osdmap e145: 8 total, 7 up, 8 in 2024-03-30T14:35:16.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-03-30T14:35:16.155 INFO:teuthology.orchestra.run.smithi031.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-30T14:35:16.155 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:35:16.156 INFO:teuthology.orchestra.run.smithi031.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-03-30T14:35:16.156 INFO:teuthology.orchestra.run.smithi031.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-30T14:35:16.256 INFO:teuthology.orchestra.run.smithi031.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-30T14:35:16.256 INFO:teuthology.orchestra.run.smithi031.stdout:releasing clients 2024-03-30T14:35:16.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:16 smithi193 podman[64290]: 2024-03-30 14:35:16.120278607 +0000 UTC m=+2.744729809 container cleanup 1d6e1da533f85104a5768c4af80339dc65a7110ec3abfac5ea44c2bbc316343f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, architecture=x86_64, distribution-scope=public, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-30T14:35:16.335 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:16 smithi193 bash[64290]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7 2024-03-30T14:35:16.624 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:16 smithi193 podman[64307]: 2024-03-30 14:35:16.338146284 +0000 UTC m=+1.022866967 container remove 1d6e1da533f85104a5768c4af80339dc65a7110ec3abfac5ea44c2bbc316343f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, io.openshift.expose-services=, RELEASE=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.19.8) 2024-03-30T14:35:16.764 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:16 smithi031 ceph-mon[67456]: osdmap e146: 8 total, 7 up, 8 in 2024-03-30T14:35:16.906 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:16 smithi193 ceph-mon[52977]: osdmap e146: 8 total, 7 up, 8 in 2024-03-30T14:35:16.906 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:16 smithi193 podman[64394]: 2024-03-30 14:35:16.570860687 +0000 UTC m=+0.023242413 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:35:16.906 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:16 smithi193 podman[64394]: 2024-03-30 14:35:16.698015645 +0000 UTC m=+0.150397350 container create 1c7955f8c63e3582293c00011c57eca19e16e8c8c5dca35bfabba62d3398b7c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:35:16.906 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:16 smithi193 podman[64394]: 2024-03-30 14:35:16.909225389 +0000 UTC m=+0.361607095 container init 1c7955f8c63e3582293c00011c57eca19e16e8c8c5dca35bfabba62d3398b7c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=) 2024-03-30T14:35:17.243 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:16 smithi193 podman[64394]: 2024-03-30 14:35:16.913774961 +0000 UTC m=+0.366156673 container start 1c7955f8c63e3582293c00011c57eca19e16e8c8c5dca35bfabba62d3398b7c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-30T14:35:17.243 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:17 smithi193 podman[64394]: 2024-03-30 14:35:17.009112946 +0000 UTC m=+0.461494656 container attach 1c7955f8c63e3582293c00011c57eca19e16e8c8c5dca35bfabba62d3398b7c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_CLEAN=True) 2024-03-30T14:35:17.585 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:17 smithi193 podman[64394]: 2024-03-30 14:35:17.246654623 +0000 UTC m=+0.699036333 container died 1c7955f8c63e3582293c00011c57eca19e16e8c8c5dca35bfabba62d3398b7c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=) 2024-03-30T14:35:17.814 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:17 smithi031 ceph-mon[67456]: pgmap v293: 97 pgs: 12 peering, 11 stale+active+clean, 74 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-03-30T14:35:17.886 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:17 smithi193 ceph-mon[52977]: pgmap v293: 97 pgs: 12 peering, 11 stale+active+clean, 74 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-03-30T14:35:18.177 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:17 smithi193 podman[64420]: 2024-03-30 14:35:17.889126823 +0000 UTC m=+0.636188362 container remove 1c7955f8c63e3582293c00011c57eca19e16e8c8c5dca35bfabba62d3398b7c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-30T14:35:18.177 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:17 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.7.service: Deactivated successfully. 2024-03-30T14:35:18.177 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:17 smithi193 systemd[1]: Stopped Ceph osd.7 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:35:18.177 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:17 smithi193 systemd[1]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.7.service: Consumed 6min 54.499s CPU time. 2024-03-30T14:35:18.257 INFO:teuthology.orchestra.run.smithi031.stdout: 5 312 22.89 MB/sec execute 1 sec latency 465.385 ms 2024-03-30T14:35:18.454 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:18 smithi193 systemd[1]: Starting Ceph osd.7 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:35:18.454 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:18 smithi193 podman[64513]: 2024-03-30 14:35:18.322042645 +0000 UTC m=+0.021888073 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:35:18.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:18 smithi193 ceph-mon[52977]: pgmap v294: 97 pgs: 5 active+undersized, 25 active+undersized+degraded, 12 peering, 55 active+clean; 359 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.8 MiB/s wr, 462 op/s; 194/1674 objects degraded (11.589%) 2024-03-30T14:35:18.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:18 smithi193 ceph-mon[52977]: Health check failed: Degraded data redundancy: 194/1674 objects degraded (11.589%), 25 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:18.810 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:18 smithi193 podman[64513]: 2024-03-30 14:35:18.457414414 +0000 UTC m=+0.157259820 container create 0d24f8eefbc332249adce119a86bd577dc95b459bbfe9cd34350faae057ede0f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-03-30T14:35:18.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:18 smithi031 ceph-mon[67456]: pgmap v294: 97 pgs: 5 active+undersized, 25 active+undersized+degraded, 12 peering, 55 active+clean; 359 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.8 MiB/s wr, 462 op/s; 194/1674 objects degraded (11.589%) 2024-03-30T14:35:18.858 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:18 smithi031 ceph-mon[67456]: Health check failed: Degraded data redundancy: 194/1674 objects degraded (11.589%), 25 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:19.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:18 smithi193 podman[64513]: 2024-03-30 14:35:18.943925664 +0000 UTC m=+0.643771077 container init 0d24f8eefbc332249adce119a86bd577dc95b459bbfe9cd34350faae057ede0f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:35:19.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:18 smithi193 podman[64513]: 2024-03-30 14:35:18.948565782 +0000 UTC m=+0.648411195 container start 0d24f8eefbc332249adce119a86bd577dc95b459bbfe9cd34350faae057ede0f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc) 2024-03-30T14:35:19.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 podman[64513]: 2024-03-30 14:35:19.035776552 +0000 UTC m=+0.735621957 container attach 0d24f8eefbc332249adce119a86bd577dc95b459bbfe9cd34350faae057ede0f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc) 2024-03-30T14:35:19.256 INFO:teuthology.orchestra.run.smithi031.stdout: 5 454 22.63 MB/sec execute 2 sec latency 58.577 ms 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate[64528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate[64528]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 bash[64513]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 bash[64513]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 bash[64513]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 bash[64513]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 bash[64513]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 bash[64513]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 bash[64513]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate[64528]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate[64528]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate[64528]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate[64528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-30T14:35:19.796 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate[64528]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-30T14:35:19.797 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:19 smithi193 podman[64513]: 2024-03-30 14:35:19.489902783 +0000 UTC m=+1.189748198 container died 0d24f8eefbc332249adce119a86bd577dc95b459bbfe9cd34350faae057ede0f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:35:19.925 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:35:20.256 INFO:teuthology.orchestra.run.smithi031.stdout: 5 715 23.99 MB/sec execute 3 sec latency 158.721 ms 2024-03-30T14:35:20.347 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 podman[64644]: 2024-03-30 14:35:20.099367097 +0000 UTC m=+0.603400717 container remove 0d24f8eefbc332249adce119a86bd577dc95b459bbfe9cd34350faae057ede0f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-30T14:35:20.347 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 podman[64708]: 2024-03-30 14:35:20.224139218 +0000 UTC m=+0.018504799 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:35:20.608 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 podman[64708]: 2024-03-30 14:35:20.350347717 +0000 UTC m=+0.144713333 container create 62b9d172403e9641505f9cfb1a555092e28b7303e73075b187b6b4c270263a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, RELEASE=squid-4e50bbc, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-30T14:35:20.608 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 podman[64708]: 2024-03-30 14:35:20.611354047 +0000 UTC m=+0.405719619 container init 62b9d172403e9641505f9cfb1a555092e28b7303e73075b187b6b4c270263a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (9m) 2m ago 18m 23.7M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (5m) 2m ago 18m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (5m) 33s ago 17m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 2m ago 18m - - 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (12m) 2m ago 12m 40.4M - 17.2.0 e1d6a67b021e 85f853686853 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (12m) 33s ago 12m 15.9M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (8m) 2m ago 19m 507M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (7m) 33s ago 17m 452M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (7m) 2m ago 20m 52.1M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (6m) 33s ago 17m 44.5M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (12m) 2m ago 12m 280M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (12m) 33s ago 12m 85.1M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (9m) 2m ago 18m 17.3M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (8m) 33s ago 17m 17.1M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (4m) 2m ago 17m 363M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (4m) 2m ago 16m 303M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (3m) 2m ago 16m 227M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (2m) 2m ago 15m 12.7M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (112s) 33s ago 15m 295M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (76s) 33s ago 14m 265M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (38s) 33s ago 14m 12.4M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 starting - - - 2704M 2024-03-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (7m) 2m ago 17m 70.0M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:35:21.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 podman[64708]: 2024-03-30 14:35:20.614529403 +0000 UTC m=+0.408894981 container start 62b9d172403e9641505f9cfb1a555092e28b7303e73075b187b6b4c270263a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-30T14:35:21.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 ceph-osd[64727]: -- 172.21.15.193:0/49861489 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eaf4c35680 con 0x55eaf4bfec00 2024-03-30T14:35:21.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 bash[64708]: 62b9d172403e9641505f9cfb1a555092e28b7303e73075b187b6b4c270263a95 2024-03-30T14:35:21.085 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:20 smithi193 systemd[1]: Started Ceph osd.7 for dc60522e-ee9f-11ee-b647-cb9ed24678a4. 2024-03-30T14:35:21.257 INFO:teuthology.orchestra.run.smithi031.stdout: 5 1000 19.25 MB/sec execute 4 sec latency 33.169 ms 2024-03-30T14:35:21.453 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:21 smithi031 ceph-mon[67456]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:21.453 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:21 smithi031 ceph-mon[67456]: pgmap v295: 97 pgs: 5 active+undersized, 29 active+undersized+degraded, 8 peering, 55 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.7 MiB/s wr, 305 op/s; 208/1587 objects degraded (13.106%) 2024-03-30T14:35:21.453 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:21 smithi031 ceph-mon[67456]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:21.453 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:21.453 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:21.453 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:21.540 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:35:21.540 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:35:21.540 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:35:21.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:21.540 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 7 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 11 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:35:21.541 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:35:21.584 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:35:21.161+0000 7fa67ec917c0 -1 Falling back to public interface 2024-03-30T14:35:21.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:21 smithi193 ceph-mon[52977]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:21.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:21 smithi193 ceph-mon[52977]: pgmap v295: 97 pgs: 5 active+undersized, 29 active+undersized+degraded, 8 peering, 55 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.7 MiB/s wr, 305 op/s; 208/1587 objects degraded (13.106%) 2024-03-30T14:35:21.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:21 smithi193 ceph-mon[52977]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:21.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:21.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:21.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:35:22.142 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "13/23 daemons upgraded", 2024-03-30T14:35:22.143 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T14:35:22.143 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:35:22.143 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:35:22.257 INFO:teuthology.orchestra.run.smithi031.stdout: 5 cleanup 5 sec 2024-03-30T14:35:22.474 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:22 smithi193 ceph-mon[52977]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:22.474 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:22 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2695761829' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:22.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:22 smithi031 ceph-mon[67456]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:22.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:22 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2695761829' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: 0 cleanup 5 sec 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Operation Count AvgLat MaxLat 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: ---------------------------------------- 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: NTCreateX 870 7.311 58.570 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Close 795 8.558 165.676 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Rename 50 11.662 43.500 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Unlink 86 3.771 22.993 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Qpathinfo 502 1.553 26.387 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Qfileinfo 385 15.240 91.194 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Qfsinfo 162 0.569 3.801 2024-03-30T14:35:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: Sfileinfo 166 13.709 25.574 2024-03-30T14:35:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: Find 233 0.967 3.220 2024-03-30T14:35:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: WriteX 1797 0.025 0.067 2024-03-30T14:35:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: ReadX 1037 0.016 0.648 2024-03-30T14:35:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: Flush 122 14.208 37.330 2024-03-30T14:35:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:35:22.500 INFO:teuthology.orchestra.run.smithi031.stdout:Throughput 19.2531 MB/sec 5 clients 5 procs max_latency=465.385 ms 2024-03-30T14:35:22.501 DEBUG:teuthology.parallel:result is None 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 225/1482 objects degraded (15.182%), 35 pgs degraded 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout: osd.7 (root=default,host=smithi193) is down 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 225/1482 objects degraded (15.182%), 35 pgs degraded 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-30T14:35:22.706 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-30T14:35:22.707 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.a is active+undersized+degraded, acting [6,1] 2024-03-30T14:35:22.708 INFO:teuthology.orchestra.run.smithi031.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-30T14:35:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:23 smithi193 ceph-mon[52977]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:23 smithi193 ceph-mon[52977]: pgmap v296: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s wr, 366 op/s; 225/1482 objects degraded (15.182%) 2024-03-30T14:35:23.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:23 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2943768600' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:35:23.611 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:23 smithi031 ceph-mon[67456]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:23.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:23 smithi031 ceph-mon[67456]: pgmap v296: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s wr, 366 op/s; 225/1482 objects degraded (15.182%) 2024-03-30T14:35:23.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:23 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2943768600' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:35:24.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:24 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 203/1404 objects degraded (14.459%), 35 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:24.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:24 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 203/1404 objects degraded (14.459%), 35 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:25.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:25 smithi193 ceph-mon[52977]: pgmap v297: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s wr, 385 op/s; 203/1404 objects degraded (14.459%) 2024-03-30T14:35:25.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:25.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:25 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:25.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:25 smithi031 ceph-mon[67456]: pgmap v297: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s wr, 385 op/s; 203/1404 objects degraded (14.459%) 2024-03-30T14:35:25.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:25.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:25 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:27.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:27.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:27.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:26 smithi193 ceph-mon[52977]: pgmap v298: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 317 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s wr, 395 op/s; 182/1257 objects degraded (14.479%) 2024-03-30T14:35:27.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:27.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:27.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:26 smithi031 ceph-mon[67456]: pgmap v298: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 317 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s wr, 395 op/s; 182/1257 objects degraded (14.479%) 2024-03-30T14:35:28.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:28.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:28.861 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:28.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:29.232 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:35:28.962+0000 7fa67ec917c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-30T14:35:29.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: pgmap v299: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 11 MiB/s wr, 375 op/s; 82/576 objects degraded (14.236%) 2024-03-30T14:35:29.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:29.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-30T14:35:29.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-30T14:35:29.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:29 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 82/576 objects degraded (14.236%), 32 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:30.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: pgmap v299: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 11 MiB/s wr, 375 op/s; 82/576 objects degraded (14.236%) 2024-03-30T14:35:30.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:30.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:30.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-30T14:35:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-30T14:35:30.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:29 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 82/576 objects degraded (14.236%), 32 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:30.835 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:35:30.375+0000 7fa67ec917c0 -1 osd.7 144 log_to_monitors true 2024-03-30T14:35:30.835 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:35:30.525+0000 7fa6700e9700 -1 osd.7 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:35:30.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: pgmap v300: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 8.8 MiB/s wr, 291 op/s; 82/576 objects degraded (14.236%) 2024-03-30T14:35:30.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: pgmap v301: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 10 MiB/s wr, 345 op/s; 82/576 objects degraded (14.236%) 2024-03-30T14:35:30.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all osd 2024-03-30T14:35:30.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: Upgrade: Setting require_osd_release to 19 squid 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: from='osd.7 [v2:172.21.15.193:6824/1183658295,v1:172.21.15.193:6825/1183658295]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: osdmap e147: 8 total, 7 up, 8 in 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: from='osd.7 [v2:172.21.15.193:6824/1183658295,v1:172.21.15.193:6825/1183658295]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:35:30.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:30 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi031.xsynqy"]}]: dispatch 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: pgmap v300: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 8.8 MiB/s wr, 291 op/s; 82/576 objects degraded (14.236%) 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: pgmap v301: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 10 MiB/s wr, 345 op/s; 82/576 objects degraded (14.236%) 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all osd 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: Upgrade: Setting require_osd_release to 19 squid 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: from='osd.7 [v2:172.21.15.193:6824/1183658295,v1:172.21.15.193:6825/1183658295]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-30T14:35:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T14:35:30.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: osdmap e147: 8 total, 7 up, 8 in 2024-03-30T14:35:30.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: from='osd.7 [v2:172.21.15.193:6824/1183658295,v1:172.21.15.193:6825/1183658295]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:35:30.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T14:35:30.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:30 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi031.xsynqy"]}]: dispatch 2024-03-30T14:35:31.687 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:31 smithi031 ceph-mon[67456]: Upgrade: It appears safe to stop mds.foofs.smithi031.xsynqy 2024-03-30T14:35:31.687 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:31 smithi031 ceph-mon[67456]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:35:31.687 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:31 smithi031 ceph-mon[67456]: osd.7 [v2:172.21.15.193:6824/1183658295,v1:172.21.15.193:6825/1183658295] boot 2024-03-30T14:35:31.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:31 smithi031 ceph-mon[67456]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:35:31.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:35:32.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:31 smithi193 ceph-mon[52977]: Upgrade: It appears safe to stop mds.foofs.smithi031.xsynqy 2024-03-30T14:35:32.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:31 smithi193 ceph-mon[52977]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:35:32.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:31 smithi193 ceph-mon[52977]: osd.7 [v2:172.21.15.193:6824/1183658295,v1:172.21.15.193:6825/1183658295] boot 2024-03-30T14:35:32.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:31 smithi193 ceph-mon[52977]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:35:32.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:35:32.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:32 smithi031 ceph-mon[67456]: pgmap v303: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 240 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 832 KiB/s wr, 146 op/s; 81/492 objects degraded (16.463%) 2024-03-30T14:35:32.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:32 smithi031 ceph-mon[67456]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:35:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:32 smithi193 ceph-mon[52977]: pgmap v303: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 240 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 832 KiB/s wr, 146 op/s; 81/492 objects degraded (16.463%) 2024-03-30T14:35:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:32 smithi193 ceph-mon[52977]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:35:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:34 smithi031 ceph-mon[67456]: Upgrade: Updating mds.foofs.smithi031.xsynqy 2024-03-30T14:35:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:35:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:34 smithi031 ceph-mon[67456]: Deploying daemon mds.foofs.smithi031.xsynqy on smithi031 2024-03-30T14:35:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:34 smithi031 ceph-mon[67456]: pgmap v306: 97 pgs: 2 active+recovery_wait+degraded, 16 peering, 3 active+undersized, 20 active+undersized+degraded, 56 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 8.2 KiB/s wr, 14 op/s; 79/441 objects degraded (17.914%); 0 B/s, 2 objects/s recovering 2024-03-30T14:35:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:34 smithi031 ceph-mon[67456]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:35:34.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:34 smithi193 ceph-mon[52977]: Upgrade: Updating mds.foofs.smithi031.xsynqy 2024-03-30T14:35:34.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:34 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:34.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:34 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi031.xsynqy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:35:34.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:34 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:34.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:34 smithi193 ceph-mon[52977]: Deploying daemon mds.foofs.smithi031.xsynqy on smithi031 2024-03-30T14:35:34.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:34 smithi193 ceph-mon[52977]: pgmap v306: 97 pgs: 2 active+recovery_wait+degraded, 16 peering, 3 active+undersized, 20 active+undersized+degraded, 56 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 8.2 KiB/s wr, 14 op/s; 79/441 objects degraded (17.914%); 0 B/s, 2 objects/s recovering 2024-03-30T14:35:34.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:34 smithi193 ceph-mon[52977]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:35:35.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:35 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 79/441 objects degraded (17.914%), 22 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:35.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:35 smithi193 ceph-mon[52977]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:35:35.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:35 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 79/441 objects degraded (17.914%), 22 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:35.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:35 smithi031 ceph-mon[67456]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: pgmap v309: 97 pgs: 2 active+recovery_wait+degraded, 16 peering, 3 active+undersized, 17 active+undersized+degraded, 59 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.5 KiB/s wr, 6 op/s; 67/441 objects degraded (15.193%); 0 B/s, 2 objects/s recovering 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: Standby daemon mds.foofs.smithi193.shgxxs assigned to filesystem foofs as rank 0 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T14:35:36.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T14:35:36.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-30T14:35:36.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:36 smithi193 ceph-mon[52977]: fsmap foofs:1/1 {0=foofs.smithi193.shgxxs=up:replay} 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[67450]: 2024-03-30T14:35:36.442+0000 7fa41f9d4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: pgmap v309: 97 pgs: 2 active+recovery_wait+degraded, 16 peering, 3 active+undersized, 17 active+undersized+degraded, 59 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.5 KiB/s wr, 6 op/s; 67/441 objects degraded (15.193%); 0 B/s, 2 objects/s recovering 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: Standby daemon mds.foofs.smithi193.shgxxs assigned to filesystem foofs as rank 0 2024-03-30T14:35:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T14:35:36.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T14:35:36.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-30T14:35:36.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:36 smithi031 ceph-mon[67456]: fsmap foofs:1/1 {0=foofs.smithi193.shgxxs=up:replay} 2024-03-30T14:35:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:38 smithi193 ceph-mon[52977]: pgmap v312: 97 pgs: 2 active+recovery_wait+degraded, 5 peering, 90 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 0 B/s wr, 4 op/s; 31/432 objects degraded (7.176%); 1022 KiB/s, 56 objects/s recovering 2024-03-30T14:35:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:38 smithi193 ceph-mon[52977]: reconnect by client.24363 172.21.15.193:0/4253260584 after 0.000999983 2024-03-30T14:35:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:38 smithi193 ceph-mon[52977]: reconnect by client.14558 172.21.15.31:0/230223430 after 0.000999983 2024-03-30T14:35:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:38 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:reconnect 2024-03-30T14:35:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:38 smithi193 ceph-mon[52977]: fsmap foofs:1/1 {0=foofs.smithi193.shgxxs=up:reconnect} 2024-03-30T14:35:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:35:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:38 smithi031 ceph-mon[67456]: pgmap v312: 97 pgs: 2 active+recovery_wait+degraded, 5 peering, 90 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 0 B/s wr, 4 op/s; 31/432 objects degraded (7.176%); 1022 KiB/s, 56 objects/s recovering 2024-03-30T14:35:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:38 smithi031 ceph-mon[67456]: reconnect by client.24363 172.21.15.193:0/4253260584 after 0.000999983 2024-03-30T14:35:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:38 smithi031 ceph-mon[67456]: reconnect by client.14558 172.21.15.31:0/230223430 after 0.000999983 2024-03-30T14:35:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:38 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:reconnect 2024-03-30T14:35:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:38 smithi031 ceph-mon[67456]: fsmap foofs:1/1 {0=foofs.smithi193.shgxxs=up:reconnect} 2024-03-30T14:35:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:35:39.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:39 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:rejoin 2024-03-30T14:35:39.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:39 smithi031 ceph-mon[67456]: fsmap foofs:1/1 {0=foofs.smithi193.shgxxs=up:rejoin} 2024-03-30T14:35:39.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:39 smithi031 ceph-mon[67456]: daemon mds.foofs.smithi193.shgxxs is now active in filesystem foofs as rank 0 2024-03-30T14:35:39.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:39 smithi031 ceph-mon[67456]: Health check update: Degraded data redundancy: 31/432 objects degraded (7.176%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:39.886 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:39 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:rejoin 2024-03-30T14:35:39.886 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:39 smithi193 ceph-mon[52977]: fsmap foofs:1/1 {0=foofs.smithi193.shgxxs=up:rejoin} 2024-03-30T14:35:39.886 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:39 smithi193 ceph-mon[52977]: daemon mds.foofs.smithi193.shgxxs is now active in filesystem foofs as rank 0 2024-03-30T14:35:39.886 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:39 smithi193 ceph-mon[52977]: Health check update: Degraded data redundancy: 31/432 objects degraded (7.176%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T14:35:40.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:40 smithi031 ceph-mon[67456]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 352 B/s wr, 6 op/s; 1.3 MiB/s, 44 objects/s recovering 2024-03-30T14:35:40.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:40 smithi031 ceph-mon[67456]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/432 objects degraded (7.176%), 2 pgs degraded) 2024-03-30T14:35:40.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:40 smithi031 ceph-mon[67456]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T14:35:40.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:40 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:active 2024-03-30T14:35:40.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:40 smithi031 ceph-mon[67456]: fsmap foofs:1 {0=foofs.smithi193.shgxxs=up:active} 2024-03-30T14:35:41.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:40 smithi193 ceph-mon[52977]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 352 B/s wr, 6 op/s; 1.3 MiB/s, 44 objects/s recovering 2024-03-30T14:35:41.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:40 smithi193 ceph-mon[52977]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/432 objects degraded (7.176%), 2 pgs degraded) 2024-03-30T14:35:41.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:40 smithi193 ceph-mon[52977]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T14:35:41.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:40 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.193:6832/776690103,v1:172.21.15.193:6833/776690103] up:active 2024-03-30T14:35:41.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:40 smithi193 ceph-mon[52977]: fsmap foofs:1 {0=foofs.smithi193.shgxxs=up:active} 2024-03-30T14:35:42.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:42 smithi031 ceph-mon[67456]: pgmap v314: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 302 B/s wr, 11 op/s; 1.1 MiB/s, 44 objects/s recovering 2024-03-30T14:35:42.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:42.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:42 smithi193 ceph-mon[52977]: pgmap v314: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 302 B/s wr, 11 op/s; 1.1 MiB/s, 44 objects/s recovering 2024-03-30T14:35:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:44.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:44 smithi031 ceph-mon[67456]: pgmap v315: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 511 B/s wr, 14 op/s; 953 KiB/s, 37 objects/s recovering 2024-03-30T14:35:45.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:44 smithi193 ceph-mon[52977]: pgmap v315: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 511 B/s wr, 14 op/s; 953 KiB/s, 37 objects/s recovering 2024-03-30T14:35:46.741 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:46 smithi031 ceph-mon[67456]: pgmap v316: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 419 B/s wr, 11 op/s; 781 KiB/s, 19 objects/s recovering 2024-03-30T14:35:47.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:46 smithi193 ceph-mon[52977]: pgmap v316: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 419 B/s wr, 11 op/s; 781 KiB/s, 19 objects/s recovering 2024-03-30T14:35:48.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:48 smithi031 ceph-mon[67456]: pgmap v317: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 376 B/s wr, 10 op/s; 325 KiB/s, 13 objects/s recovering 2024-03-30T14:35:48.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:48 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:48.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:48 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:48.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:48 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:48.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:48 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:49.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:48 smithi193 ceph-mon[52977]: pgmap v317: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 376 B/s wr, 10 op/s; 325 KiB/s, 13 objects/s recovering 2024-03-30T14:35:49.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:48 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:49.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:48 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:49.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:48 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:49.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:48 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:49.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:49 smithi031 ceph-mon[67456]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T14:35:49.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:49 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:boot 2024-03-30T14:35:49.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:49 smithi031 ceph-mon[67456]: fsmap foofs:1 {0=foofs.smithi193.shgxxs=up:active} 1 up:standby 2024-03-30T14:35:49.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:49 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:35:50.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:49 smithi193 ceph-mon[52977]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T14:35:50.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:49 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:boot 2024-03-30T14:35:50.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:49 smithi193 ceph-mon[52977]: fsmap foofs:1 {0=foofs.smithi193.shgxxs=up:active} 1 up:standby 2024-03-30T14:35:50.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:49 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:35:50.870 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:50 smithi193 ceph-mon[52977]: pgmap v318: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 7.3 KiB/s wr, 11 op/s; 294 KiB/s, 6 objects/s recovering 2024-03-30T14:35:50.912 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:50 smithi031 ceph-mon[67456]: pgmap v318: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 7.3 KiB/s wr, 11 op/s; 294 KiB/s, 6 objects/s recovering 2024-03-30T14:35:52.910 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:52 smithi031 ceph-mon[67456]: pgmap v319: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s; 0 B/s, 3 objects/s recovering 2024-03-30T14:35:53.071 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:35:53.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:52 smithi193 ceph-mon[52977]: pgmap v319: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s; 0 B/s, 3 objects/s recovering 2024-03-30T14:35:53.756 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (10m) 0s ago 19m 23.9M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (6m) 0s ago 19m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (6m) 28s ago 18m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 0s ago 18m - - 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (5s) 0s ago 13m 13.6M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 5aea88780a6b 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (13m) 28s ago 13m 16.0M - 17.2.0 e1d6a67b021e 6f925d598d00 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (8m) 0s ago 20m 512M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (8m) 28s ago 18m 452M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (7m) 0s ago 20m 63.9M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (7m) 28s ago 18m 46.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (12m) 0s ago 12m 279M - 4.0 e1d6a67b021e 02e09c0aa922 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (12m) 28s ago 12m 85.2M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (9m) 0s ago 18m 17.5M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (9m) 28s ago 18m 15.5M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:35:53.757 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (5m) 0s ago 17m 430M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (4m) 0s ago 17m 397M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (3m) 0s ago 16m 370M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (3m) 0s ago 16m 414M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (2m) 28s ago 15m 339M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (108s) 28s ago 15m 284M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (71s) 28s ago 15m 280M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (33s) 28s ago 14m 15.1M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 62b9d172403e 2024-03-30T14:35:53.758 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (8m) 0s ago 17m 72.2M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:35:54.169 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:35:54.169 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:35:54.169 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:35:54.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 8 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 1 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 13 2024-03-30T14:35:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:35:54.171 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "osd", 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr" 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "15/23 daemons upgraded", 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T14:35:54.532 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:35:54.533 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:35:54.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:54 smithi031 ceph-mon[67456]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:54.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:54.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:54 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:54.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:54 smithi031 ceph-mon[67456]: pgmap v320: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-30T14:35:54.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:54 smithi031 ceph-mon[67456]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:54.533 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:54 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/763307576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:54.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:54 smithi193 ceph-mon[52977]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:54.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:54.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:54 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:54.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:54 smithi193 ceph-mon[52977]: pgmap v320: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-30T14:35:54.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:54 smithi193 ceph-mon[52977]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:54.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:54 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/763307576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:54.967 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:35:54.967 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:35:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:35:55.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:55 smithi031 ceph-mon[67456]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:55.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:55.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:55 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:55.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:55 smithi031 ceph-mon[67456]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:55.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:55 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2302197362' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:35:55.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:55 smithi193 ceph-mon[52977]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:55.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:55.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:55 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:55.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:55 smithi193 ceph-mon[52977]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:35:55.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:55 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2302197362' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:35:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:56 smithi193 ceph-mon[52977]: pgmap v321: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-30T14:35:56.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:56 smithi031 ceph-mon[67456]: pgmap v321: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-30T14:35:58.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: pgmap v322: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-30T14:35:58.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:58.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi193.shgxxs"]}]: dispatch 2024-03-30T14:35:58.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: pgmap v322: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-30T14:35:58.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:35:58.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.952 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:35:58.953 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi193.shgxxs"]}]: dispatch 2024-03-30T14:35:59.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:59 smithi031 ceph-mon[67456]: Detected new or changed devices on smithi031 2024-03-30T14:35:59.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:59 smithi031 ceph-mon[67456]: pgmap v323: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 7.9 KiB/s wr, 3 op/s 2024-03-30T14:35:59.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:59 smithi031 ceph-mon[67456]: pgmap v324: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 229 B/s rd, 229 B/s wr, 0 op/s 2024-03-30T14:35:59.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:35:59 smithi031 ceph-mon[67456]: Upgrade: It appears safe to stop mds.foofs.smithi193.shgxxs 2024-03-30T14:35:59.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:59 smithi193 ceph-mon[52977]: Detected new or changed devices on smithi031 2024-03-30T14:35:59.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:59 smithi193 ceph-mon[52977]: pgmap v323: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 7.9 KiB/s wr, 3 op/s 2024-03-30T14:35:59.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:59 smithi193 ceph-mon[52977]: pgmap v324: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 229 B/s rd, 229 B/s wr, 0 op/s 2024-03-30T14:35:59.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:35:59 smithi193 ceph-mon[52977]: Upgrade: It appears safe to stop mds.foofs.smithi193.shgxxs 2024-03-30T14:36:01.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:00 smithi193 ceph-mon[52977]: Upgrade: Updating mds.foofs.smithi193.shgxxs 2024-03-30T14:36:01.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:01.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:36:01.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:01.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:00 smithi193 ceph-mon[52977]: Deploying daemon mds.foofs.smithi193.shgxxs on smithi193 2024-03-30T14:36:01.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:00 smithi193 ceph-mon[52977]: pgmap v325: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 229 B/s wr, 0 op/s 2024-03-30T14:36:01.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:00 smithi031 ceph-mon[67456]: Upgrade: Updating mds.foofs.smithi193.shgxxs 2024-03-30T14:36:01.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:01.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.shgxxs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T14:36:01.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:01.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:00 smithi031 ceph-mon[67456]: Deploying daemon mds.foofs.smithi193.shgxxs on smithi193 2024-03-30T14:36:01.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:00 smithi031 ceph-mon[67456]: pgmap v325: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 229 B/s wr, 0 op/s 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: Standby daemon mds.foofs.smithi031.xsynqy assigned to filesystem foofs as rank 0 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-30T14:36:02.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:01 smithi193 ceph-mon[52977]: fsmap foofs:1/1 {0=foofs.smithi031.xsynqy=up:replay} 2024-03-30T14:36:02.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[67450]: 2024-03-30T14:36:01.902+0000 7fa41f9d4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:36:02.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T14:36:02.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T14:36:02.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:36:02.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: Standby daemon mds.foofs.smithi031.xsynqy assigned to filesystem foofs as rank 0 2024-03-30T14:36:02.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T14:36:02.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T14:36:02.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-30T14:36:02.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:01 smithi031 ceph-mon[67456]: fsmap foofs:1/1 {0=foofs.smithi031.xsynqy=up:replay} 2024-03-30T14:36:03.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:02 smithi193 ceph-mon[52977]: pgmap v327: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 296 B/s rd, 296 B/s wr, 0 op/s 2024-03-30T14:36:03.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:02 smithi031 ceph-mon[67456]: pgmap v327: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 296 B/s rd, 296 B/s wr, 0 op/s 2024-03-30T14:36:04.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:03 smithi193 ceph-mon[52977]: reconnect by client.14558 172.21.15.31:0/230223430 after 0 2024-03-30T14:36:04.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:03 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:reconnect 2024-03-30T14:36:04.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:03 smithi193 ceph-mon[52977]: fsmap foofs:1/1 {0=foofs.smithi031.xsynqy=up:reconnect} 2024-03-30T14:36:04.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:03 smithi193 ceph-mon[52977]: reconnect by client.24363 172.21.15.193:0/4253260584 after 0.00999986 2024-03-30T14:36:04.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:03 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:36:04.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:03 smithi031 ceph-mon[67456]: reconnect by client.14558 172.21.15.31:0/230223430 after 0 2024-03-30T14:36:04.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:03 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:reconnect 2024-03-30T14:36:04.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:03 smithi031 ceph-mon[67456]: fsmap foofs:1/1 {0=foofs.smithi031.xsynqy=up:reconnect} 2024-03-30T14:36:04.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:03 smithi031 ceph-mon[67456]: reconnect by client.24363 172.21.15.193:0/4253260584 after 0.00999986 2024-03-30T14:36:04.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:03 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:36:05.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:04 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:rejoin 2024-03-30T14:36:05.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:04 smithi193 ceph-mon[52977]: fsmap foofs:1/1 {0=foofs.smithi031.xsynqy=up:rejoin} 2024-03-30T14:36:05.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:04 smithi193 ceph-mon[52977]: daemon mds.foofs.smithi031.xsynqy is now active in filesystem foofs as rank 0 2024-03-30T14:36:05.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:04 smithi193 ceph-mon[52977]: pgmap v328: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 296 B/s wr, 2 op/s 2024-03-30T14:36:05.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:04 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:rejoin 2024-03-30T14:36:05.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:04 smithi031 ceph-mon[67456]: fsmap foofs:1/1 {0=foofs.smithi031.xsynqy=up:rejoin} 2024-03-30T14:36:05.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:04 smithi031 ceph-mon[67456]: daemon mds.foofs.smithi031.xsynqy is now active in filesystem foofs as rank 0 2024-03-30T14:36:05.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:04 smithi031 ceph-mon[67456]: pgmap v328: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 296 B/s wr, 2 op/s 2024-03-30T14:36:06.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:05 smithi193 ceph-mon[52977]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T14:36:06.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:05 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:active 2024-03-30T14:36:06.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:05 smithi193 ceph-mon[52977]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 2024-03-30T14:36:06.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:05 smithi031 ceph-mon[67456]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T14:36:06.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:05 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356] up:active 2024-03-30T14:36:06.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:05 smithi031 ceph-mon[67456]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 2024-03-30T14:36:07.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:06 smithi193 ceph-mon[52977]: pgmap v329: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6 op/s 2024-03-30T14:36:07.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:06 smithi031 ceph-mon[67456]: pgmap v329: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6 op/s 2024-03-30T14:36:09.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:09 smithi193 ceph-mon[52977]: pgmap v330: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 10 op/s 2024-03-30T14:36:09.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:09 smithi031 ceph-mon[67456]: pgmap v330: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 10 op/s 2024-03-30T14:36:11.377 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:11 smithi193 ceph-mon[52977]: pgmap v331: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-30T14:36:11.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:11 smithi031 ceph-mon[67456]: pgmap v331: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-30T14:36:12.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:12.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:12.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:12.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:12 smithi193 ceph-mon[52977]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T14:36:12.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:12 smithi193 ceph-mon[52977]: mds.? [v2:172.21.15.193:6832/3908827208,v1:172.21.15.193:6833/3908827208] up:boot 2024-03-30T14:36:12.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:12 smithi193 ceph-mon[52977]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:36:12.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:36:12.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:12.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:12.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:12.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:12 smithi031 ceph-mon[67456]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T14:36:12.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:12 smithi031 ceph-mon[67456]: mds.? [v2:172.21.15.193:6832/3908827208,v1:172.21.15.193:6833/3908827208] up:boot 2024-03-30T14:36:12.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:12 smithi031 ceph-mon[67456]: fsmap foofs:1 {0=foofs.smithi031.xsynqy=up:active} 1 up:standby 2024-03-30T14:36:12.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:36:13.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:13 smithi193 ceph-mon[52977]: pgmap v332: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 197 B/s wr, 10 op/s 2024-03-30T14:36:13.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:13.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:13 smithi031 ceph-mon[67456]: pgmap v332: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 197 B/s wr, 10 op/s 2024-03-30T14:36:13.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:15 smithi193 ceph-mon[52977]: pgmap v333: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-30T14:36:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:16.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:15 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:16.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:15 smithi031 ceph-mon[67456]: pgmap v333: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-30T14:36:16.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:16.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:15 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:16 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:16 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:17.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:16 smithi193 ceph-mon[52977]: pgmap v334: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-03-30T14:36:17.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:16 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:17.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:16 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:17.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:16 smithi031 ceph-mon[67456]: pgmap v334: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-03-30T14:36:19.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: pgmap v335: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-03-30T14:36:19.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:19.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:36:19.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:36:19.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi031.xsynqy"}]': finished 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi193.shgxxs"}]': finished 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: pgmap v335: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-03-30T14:36:19.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi031.xsynqy"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi031.xsynqy"}]': finished 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi193.shgxxs"}]: dispatch 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi193.shgxxs"}]': finished 2024-03-30T14:36:19.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:19.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: Detected new or changed devices on smithi193 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: pgmap v336: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 470 B/s rd, 7.8 KiB/s wr, 3 op/s 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: pgmap v337: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 9.6 KiB/s wr, 4 op/s 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all mds 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all rgw 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all rbd-mirror 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all ceph-exporter 2024-03-30T14:36:20.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:20 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all iscsi 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: Detected new or changed devices on smithi193 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: pgmap v336: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 470 B/s rd, 7.8 KiB/s wr, 3 op/s 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: pgmap v337: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 9.6 KiB/s wr, 4 op/s 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all mds 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all rgw 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all rbd-mirror 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all ceph-exporter 2024-03-30T14:36:20.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:20 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all iscsi 2024-03-30T14:36:22.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: Upgrade: Updating nfs.foo.0.0.smithi031.juvmqh (1/2) 2024-03-30T14:36:22.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:22.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: Rados config object exists: conf-nfs.foo 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh-rgw 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: Bind address in nfs.foo.0.0.smithi031.juvmqh's ganesha conf is defaulting to empty 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: Deploying daemon nfs.foo.0.0.smithi031.juvmqh on smithi031 2024-03-30T14:36:22.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:21 smithi193 ceph-mon[52977]: pgmap v338: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 9.6 KiB/s wr, 4 op/s 2024-03-30T14:36:22.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: Upgrade: Updating nfs.foo.0.0.smithi031.juvmqh (1/2) 2024-03-30T14:36:22.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:22.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh 2024-03-30T14:36:22.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: Rados config object exists: conf-nfs.foo 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: Creating key for client.nfs.foo.0.0.smithi031.juvmqh-rgw 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi031.juvmqh-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: Bind address in nfs.foo.0.0.smithi031.juvmqh's ganesha conf is defaulting to empty 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: Deploying daemon nfs.foo.0.0.smithi031.juvmqh on smithi031 2024-03-30T14:36:22.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:21 smithi031 ceph-mon[67456]: pgmap v338: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 9.6 KiB/s wr, 4 op/s 2024-03-30T14:36:24.435 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:24 smithi031 ceph-mon[67456]: pgmap v339: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 230 B/s wr, 0 op/s 2024-03-30T14:36:24.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:24 smithi193 ceph-mon[52977]: pgmap v339: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 230 B/s wr, 0 op/s 2024-03-30T14:36:25.336 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (10m) 32s ago 19m 23.9M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (6m) 32s ago 19m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (6m) 11s ago 19m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 32s ago 19m - - 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (38s) 32s ago 13m 13.6M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 5aea88780a6b 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (15s) 11s ago 13m 11.2M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 8b29ef7503d9 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (9m) 32s ago 20m 512M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (8m) 11s ago 18m 453M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:36:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (8m) 32s ago 21m 63.9M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (7m) 11s ago 18m 46.8M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 starting - - - - 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (13m) 11s ago 13m 85.2M - 4.0 e1d6a67b021e 5db1d427e246 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (10m) 32s ago 19m 17.5M - 1.3.1 1dbe0e931976 968b57f01668 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (9m) 11s ago 18m 14.9M - 1.3.1 1dbe0e931976 797080e68cb1 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (6m) 32s ago 18m 430M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (5m) 32s ago 17m 397M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (4m) 32s ago 17m 370M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (3m) 32s ago 16m 414M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (2m) 11s ago 16m 342M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (2m) 11s ago 16m 288M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (103s) 11s ago 15m 282M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (65s) 11s ago 15m 286M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 62b9d172403e 2024-03-30T14:36:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (8m) 32s ago 18m 72.2M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:36:26.233 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:26 smithi031 ceph-mon[67456]: pgmap v340: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 230 B/s wr, 0 op/s 2024-03-30T14:36:26.233 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:26 smithi031 ceph-mon[67456]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:26.233 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:26 smithi031 ceph-mon[67456]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.31:0/230223430"}]: dispatch 2024-03-30T14:36:26.234 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:26.234 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:26 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:26.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:26 smithi193 ceph-mon[52977]: pgmap v340: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 230 B/s rd, 230 B/s wr, 0 op/s 2024-03-30T14:36:26.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:26 smithi193 ceph-mon[52977]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:26.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:26 smithi193 ceph-mon[52977]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.31:0/230223430"}]: dispatch 2024-03-30T14:36:26.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:26.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:26 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 8 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:36:26.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:26.487 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:36:26.487 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 14 2024-03-30T14:36:26.487 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:36:26.487 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr", 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "mds", 2024-03-30T14:36:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "osd" 2024-03-30T14:36:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:36:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "16/23 daemons upgraded", 2024-03-30T14:36:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading nfs daemons", 2024-03-30T14:36:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:36:26.882 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:36:27.313 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:36:27.313 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:36:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:36:27.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:27 smithi193 ceph-mon[52977]: Evicting (and blocklisting) client session 14558 (172.21.15.31:0/230223430) 2024-03-30T14:36:27.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:27 smithi193 ceph-mon[52977]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:27.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:27 smithi193 ceph-mon[52977]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:27.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:27 smithi193 ceph-mon[52977]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.31:0/230223430"}]': finished 2024-03-30T14:36:27.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:27 smithi193 ceph-mon[52977]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:36:27.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:27 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1564784353' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:27.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:27 smithi031 ceph-mon[67456]: Evicting (and blocklisting) client session 14558 (172.21.15.31:0/230223430) 2024-03-30T14:36:27.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:27 smithi031 ceph-mon[67456]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:27.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:27 smithi031 ceph-mon[67456]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:27.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:27 smithi031 ceph-mon[67456]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.31:0/230223430"}]': finished 2024-03-30T14:36:27.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:27 smithi031 ceph-mon[67456]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:36:27.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:27 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1564784353' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:28.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:28.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: pgmap v342: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-30T14:36:28.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: Upgrade: Updating nfs.foo.1.0.smithi193.odcsji (2/2) 2024-03-30T14:36:28.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: Creating key for client.nfs.foo.1.0.smithi193.odcsji 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1221670014' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: Rados config object exists: conf-nfs.foo 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: Creating key for client.nfs.foo.1.0.smithi193.odcsji-rgw 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: Bind address in nfs.foo.1.0.smithi193.odcsji's ganesha conf is defaulting to empty 2024-03-30T14:36:28.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:28.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: Deploying daemon nfs.foo.1.0.smithi193.odcsji on smithi193 2024-03-30T14:36:28.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:28 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:28.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:28.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: pgmap v342: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-30T14:36:28.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: Upgrade: Updating nfs.foo.1.0.smithi193.odcsji (2/2) 2024-03-30T14:36:28.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:28.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: Creating key for client.nfs.foo.1.0.smithi193.odcsji 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1221670014' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: Rados config object exists: conf-nfs.foo 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: Creating key for client.nfs.foo.1.0.smithi193.odcsji-rgw 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.odcsji-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: Bind address in nfs.foo.1.0.smithi193.odcsji's ganesha conf is defaulting to empty 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:28.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: Deploying daemon nfs.foo.1.0.smithi193.odcsji on smithi193 2024-03-30T14:36:28.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:28 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:30.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:30 smithi193 ceph-mon[52977]: pgmap v343: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-30T14:36:30.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:30 smithi031 ceph-mon[67456]: pgmap v343: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-30T14:36:32.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:32 smithi193 ceph-mon[52977]: pgmap v344: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-30T14:36:32.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:32.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:32.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:32 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:32.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:32 smithi031 ceph-mon[67456]: pgmap v344: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-30T14:36:32.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:32.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:32.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:32 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:34.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:34 smithi193 ceph-mon[52977]: pgmap v345: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:34.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:34 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:34.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:34 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:34.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:34 smithi193 ceph-mon[52977]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.193:0/4253260584"}]: dispatch 2024-03-30T14:36:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:34 smithi031 ceph-mon[67456]: pgmap v345: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:34 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:34.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:34 smithi031 ceph-mon[67456]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.193:0/4253260584"}]: dispatch 2024-03-30T14:36:35.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:35 smithi193 ceph-mon[52977]: Evicting (and blocklisting) client session 24363 (172.21.15.193:0/4253260584) 2024-03-30T14:36:35.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:35 smithi193 ceph-mon[52977]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.193:0/4253260584"}]': finished 2024-03-30T14:36:35.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:35 smithi193 ceph-mon[52977]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:36:35.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:35.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:35 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:35.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:35 smithi031 ceph-mon[67456]: Evicting (and blocklisting) client session 24363 (172.21.15.193:0/4253260584) 2024-03-30T14:36:35.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:35 smithi031 ceph-mon[67456]: from='mds.? [v2:172.21.15.31:6834/831713356,v1:172.21.15.31:6835/831713356]' entity='mds.foofs.smithi031.xsynqy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.193:0/4253260584"}]': finished 2024-03-30T14:36:35.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:35 smithi031 ceph-mon[67456]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:36:35.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:35.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:35 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:36.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:36 smithi031 ceph-mon[67456]: pgmap v347: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 915 B/s rd, 915 B/s wr, 2 op/s 2024-03-30T14:36:36.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:36 smithi193 ceph-mon[52977]: pgmap v347: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 915 B/s rd, 915 B/s wr, 2 op/s 2024-03-30T14:36:37.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:37 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:37.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:37 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:37.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:37 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:37.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:37 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:38 smithi193 ceph-mon[52977]: pgmap v348: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-30T14:36:38.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:38 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:38 smithi031 ceph-mon[67456]: pgmap v348: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-30T14:36:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:38.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:38 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:40.626 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:40 smithi031 ceph-mon[67456]: pgmap v349: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:40.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:40 smithi193 ceph-mon[52977]: pgmap v349: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:42.476 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: pgmap v350: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-30T14:36:42.476 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.476 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.476 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: pgmap v351: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: pgmap v352: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.477 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all nfs 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi031.juvmqh"}]: dispatch 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi031.juvmqh"}]': finished 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi193.odcsji"}]: dispatch 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi193.odcsji"}]': finished 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: Upgrade: Setting container_image for all nvmeof 2024-03-30T14:36:42.478 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: pgmap v350: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: pgmap v351: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: pgmap v352: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all nfs 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi031.juvmqh"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi031.juvmqh"}]': finished 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi193.odcsji"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi193.odcsji"}]': finished 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: Upgrade: Setting container_image for all nvmeof 2024-03-30T14:36:42.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:43.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:43.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:44.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:44 smithi193 ceph-mon[52977]: Upgrade: Updating node-exporter.smithi031 (1/2) 2024-03-30T14:36:44.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:44 smithi193 ceph-mon[52977]: Deploying daemon node-exporter.smithi031 on smithi031 2024-03-30T14:36:44.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:44 smithi193 ceph-mon[52977]: pgmap v353: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:44.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:44 smithi031 ceph-mon[67456]: Upgrade: Updating node-exporter.smithi031 (1/2) 2024-03-30T14:36:44.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:44 smithi031 ceph-mon[67456]: Deploying daemon node-exporter.smithi031 on smithi031 2024-03-30T14:36:44.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:44 smithi031 ceph-mon[67456]: pgmap v353: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:36:46.750 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:46 smithi031 ceph-mon[67456]: pgmap v354: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 494 B/s wr, 0 op/s 2024-03-30T14:36:46.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:46 smithi193 ceph-mon[52977]: pgmap v354: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 494 B/s wr, 0 op/s 2024-03-30T14:36:48.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:48 smithi193 ceph-mon[52977]: pgmap v355: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 247 B/s rd, 0 op/s 2024-03-30T14:36:48.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:48 smithi031 ceph-mon[67456]: pgmap v355: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 247 B/s rd, 0 op/s 2024-03-30T14:36:50.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:50 smithi193 ceph-mon[52977]: pgmap v356: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 247 B/s rd, 0 op/s 2024-03-30T14:36:50.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:50 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:50.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:50 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:50.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:50 smithi031 ceph-mon[67456]: pgmap v356: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 247 B/s rd, 0 op/s 2024-03-30T14:36:50.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:50 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:50.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:50 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:52.579 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:52 smithi193 ceph-mon[52977]: pgmap v357: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:36:52.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:52 smithi193 ceph-mon[52977]: Upgrade: Updating node-exporter.smithi193 (2/2) 2024-03-30T14:36:52.580 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:52 smithi193 ceph-mon[52977]: Deploying daemon node-exporter.smithi193 on smithi193 2024-03-30T14:36:52.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:52 smithi031 ceph-mon[67456]: pgmap v357: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:36:52.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:52 smithi031 ceph-mon[67456]: Upgrade: Updating node-exporter.smithi193 (2/2) 2024-03-30T14:36:52.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:52 smithi031 ceph-mon[67456]: Deploying daemon node-exporter.smithi193 on smithi193 2024-03-30T14:36:54.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:54 smithi193 ceph-mon[52977]: pgmap v358: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:36:54.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:54 smithi031 ceph-mon[67456]: pgmap v358: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:36:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:56 smithi193 ceph-mon[52977]: pgmap v359: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:36:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:56 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:56 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:56.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:56 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:56.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:56 smithi031 ceph-mon[67456]: pgmap v359: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:36:56.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:56 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:56.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:56 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:56.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:56 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:36:57.704 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:36:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:57 smithi031 ceph-mon[67456]: pgmap v360: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:36:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:58.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:58.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:36:58.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:57 smithi193 ceph-mon[52977]: pgmap v360: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:36:58.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:36:58.461 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (11m) 22s ago 20m 24.0M - 0.23.0 ba2b418f427c 93293a6f369e 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (7m) 22s ago 20m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (7m) 24s ago 19m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 22s ago 19m - - 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (70s) 22s ago 14m 35.2M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 5aea88780a6b 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (47s) 24s ago 14m 12.4M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 8b29ef7503d9 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (10m) 22s ago 21m 513M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (9m) 24s ago 19m 453M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (8m) 22s ago 21m 62.1M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (8m) 24s ago 19m 48.0M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (33s) 22s ago 13m 57.2M - 5.7 b0c4478f1d83 c23318cf4683 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (27s) 24s ago 13m 17.1M - 5.7 b0c4478f1d83 30be1448d55c 2024-03-30T14:36:58.462 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 starting - - - - 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 starting - - - - 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (6m) 22s ago 18m 432M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (5m) 22s ago 18m 398M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (5m) 22s ago 17m 373M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (4m) 22s ago 17m 416M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (3m) 24s ago 16m 343M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (2m) 24s ago 16m 288M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (2m) 24s ago 16m 283M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (98s) 24s ago 15m 285M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 62b9d172403e 2024-03-30T14:36:58.463 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (9m) 22s ago 19m 72.2M - 2.33.4 514e6a882f6e d7ce1b800494 2024-03-30T14:36:58.888 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 8 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:36:58.889 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 14 2024-03-30T14:36:58.890 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:36:58.890 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:36:59.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:58 smithi193 ceph-mon[52977]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:59.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:58 smithi193 ceph-mon[52977]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:59.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:58 smithi193 ceph-mon[52977]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:59.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:36:58 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/4023836275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:59.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:58 smithi031 ceph-mon[67456]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:59.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:58 smithi031 ceph-mon[67456]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:59.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:58 smithi031 ceph-mon[67456]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:36:59.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:36:58 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/4023836275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:36:59.291 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:36:59.291 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:36:59.291 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:36:59.291 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "mds", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "osd", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs" 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "18/23 daemons upgraded", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:36:59.292 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:36:59.746 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:36:59.746 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:36:59.746 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:37:00.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:00 smithi031 ceph-mon[67456]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:00.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:00.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:00 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:00.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:00 smithi031 ceph-mon[67456]: pgmap v361: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:00.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:00 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/4072917328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:37:00.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:00 smithi193 ceph-mon[52977]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:00.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:00.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:00 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:00.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:00 smithi193 ceph-mon[52977]: pgmap v361: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:00.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:00 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/4072917328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:37:01.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:01.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:01 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:01.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:01.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:01 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:02.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:02 smithi193 ceph-mon[52977]: pgmap v362: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:02.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:02 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:02.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:02 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:02.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:02 smithi031 ceph-mon[67456]: pgmap v362: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:02.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:02 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:02.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:02 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:04 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:04 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:04 smithi193 ceph-mon[52977]: pgmap v363: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:04.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:04 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:04 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:04 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:04 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:04 smithi031 ceph-mon[67456]: pgmap v363: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:04.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:04 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:04.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:04 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:06.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:06 smithi193 ceph-mon[52977]: pgmap v364: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:06.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:06 smithi031 ceph-mon[67456]: pgmap v364: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: pgmap v365: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 B/s rd, 0 op/s 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: pgmap v366: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:07.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.614 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:07 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: pgmap v365: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 367 B/s rd, 0 op/s 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: pgmap v366: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:07.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:07 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:09.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:09 smithi031 ceph-mon[67456]: Upgrade: Updating prometheus.smithi031 2024-03-30T14:37:09.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:09 smithi031 ceph-mon[67456]: pgmap v367: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:09.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:09 smithi031 ceph-mon[67456]: Deploying daemon prometheus.smithi031 on smithi031 2024-03-30T14:37:10.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:09 smithi193 ceph-mon[52977]: Upgrade: Updating prometheus.smithi031 2024-03-30T14:37:10.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:09 smithi193 ceph-mon[52977]: pgmap v367: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:10.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:09 smithi193 ceph-mon[52977]: Deploying daemon prometheus.smithi031 on smithi031 2024-03-30T14:37:11.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:11 smithi031 ceph-mon[67456]: pgmap v368: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:11.888 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:11 smithi193 ceph-mon[52977]: pgmap v368: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:13.662 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:13 smithi031 ceph-mon[67456]: pgmap v369: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:13.662 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:13 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:14.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:13 smithi193 ceph-mon[52977]: pgmap v369: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:14.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:13 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:15.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:14 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:15.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:14 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:15.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:14 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:15.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:14 smithi193 ceph-mon[52977]: pgmap v370: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:15.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:14 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:15.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:14 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:15.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:14 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:15.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:14 smithi031 ceph-mon[67456]: pgmap v370: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 224 B/s rd, 0 op/s 2024-03-30T14:37:17.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:17 smithi031 ceph-mon[67456]: pgmap v371: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:37:18.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:17 smithi193 ceph-mon[52977]: pgmap v371: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:37:19.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:19 smithi193 ceph-mon[52977]: pgmap v372: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:19.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:19.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:19 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:19.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:19 smithi031 ceph-mon[67456]: pgmap v372: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:19.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:19.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:19 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:21.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:21.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:20 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:21.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:20 smithi193 ceph-mon[52977]: pgmap v373: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:21.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:21.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:20 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:21.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:20 smithi031 ceph-mon[67456]: pgmap v373: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: pgmap v374: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:37:23.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:23 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: pgmap v374: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:23.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:23 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:24.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:24 smithi031 ceph-mon[67456]: pgmap v375: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-03-30T14:37:24.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:24 smithi031 ceph-mon[67456]: pgmap v376: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:24.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:24 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:37:25.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:24 smithi193 ceph-mon[52977]: pgmap v375: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-03-30T14:37:25.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:24 smithi193 ceph-mon[52977]: pgmap v376: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:25.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:24 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:37:26.634 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:26 smithi031 ceph-mon[67456]: Upgrade: Updating alertmanager.smithi031 2024-03-30T14:37:26.634 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:26 smithi031 ceph-mon[67456]: Deploying daemon alertmanager.smithi031 on smithi031 2024-03-30T14:37:26.634 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:26 smithi031 ceph-mon[67456]: pgmap v377: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:37:27.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:26 smithi193 ceph-mon[52977]: Upgrade: Updating alertmanager.smithi031 2024-03-30T14:37:27.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:26 smithi193 ceph-mon[52977]: Deploying daemon alertmanager.smithi031 on smithi031 2024-03-30T14:37:27.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:26 smithi193 ceph-mon[52977]: pgmap v377: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-03-30T14:37:27.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:28.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:27 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:28.815 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:28 smithi031 ceph-mon[67456]: pgmap v378: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:29.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:28 smithi193 ceph-mon[52977]: pgmap v378: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:30.161 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:37:30.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:30 smithi031 ceph-mon[67456]: pgmap v379: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:30.882 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 starting - - - - 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (7m) 12s ago 20m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (7m) 31s ago 20m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 error 12s ago 20m - - 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (103s) 12s ago 14m 35.5M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 5aea88780a6b 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (79s) 31s ago 14m 12.5M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 8b29ef7503d9 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (10m) 12s ago 21m 516M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (9m) 31s ago 20m 453M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (9m) 12s ago 22m 63.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (8m) 31s ago 19m 49.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (66s) 12s ago 14m 57.9M - 5.7 b0c4478f1d83 c23318cf4683 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (60s) 31s ago 14m 59.6M - 5.7 b0c4478f1d83 30be1448d55c 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (41s) 12s ago 20m 7411k - 1.5.0 0da6a335fe13 0422feae07d9 2024-03-30T14:37:30.883 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (34s) 31s ago 19m 5490k - 1.5.0 0da6a335fe13 1caae78a5321 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (7m) 12s ago 19m 433M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (6m) 12s ago 18m 399M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (5m) 12s ago 18m 374M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (4m) 12s ago 17m 417M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (4m) 31s ago 17m 344M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (3m) 31s ago 17m 288M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (2m) 31s ago 16m 283M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (2m) 31s ago 16m 285M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 62b9d172403e 2024-03-30T14:37:30.884 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (17s) 12s ago 19m 40.4M - 2.43.0 a07b618ecd1d c7e887b6e104 2024-03-30T14:37:31.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:30 smithi193 ceph-mon[52977]: pgmap v379: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:31.365 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:37:31.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:37:31.365 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:37:31.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:37:31.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:37:31.365 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:37:31.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 8 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 14 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:37:31.366 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:37:31.736 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:37:31.736 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:37:31.736 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": true, 2024-03-30T14:37:31.736 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T14:37:31.736 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [ 2024-03-30T14:37:31.736 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "mon", 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr", 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "mds", 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "osd", 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs" 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": "18/23 daemons upgraded", 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "Currently upgrading alertmanager daemons", 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:37:31.737 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:37:32.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:31 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/23667890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:32.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:32.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:32.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:32.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:32.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:31 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/23667890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:32.246 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-30T14:37:32.246 INFO:teuthology.orchestra.run.smithi031.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-30T14:37:32.246 INFO:teuthology.orchestra.run.smithi031.stdout: daemon grafana.smithi031 on smithi031 is in error state 2024-03-30T14:37:32.947 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:32 smithi031 ceph-mon[67456]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:32.947 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:32 smithi031 ceph-mon[67456]: pgmap v380: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:32.947 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:32 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/4129119890' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:37:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:32 smithi193 ceph-mon[52977]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:32 smithi193 ceph-mon[52977]: pgmap v380: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-03-30T14:37:33.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:32 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/4129119890' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:37:33.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:33 smithi031 ceph-mon[67456]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:34.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:33 smithi193 ceph-mon[52977]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:37:35.054 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:34 smithi031 ceph-mon[67456]: pgmap v381: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:37:35.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:34 smithi193 ceph-mon[52977]: pgmap v381: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:37:36.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:36 smithi193 ceph-mon[52977]: pgmap v382: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:36.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:36 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:36.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:36 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:36 smithi031 ceph-mon[67456]: pgmap v382: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:36 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:36.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:36 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:38.044 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:37 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:38.045 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:37 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:38.045 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:37 smithi031 ceph-mon[67456]: pgmap v383: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:37:38.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:37 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:38.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:37 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:38.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:37 smithi193 ceph-mon[52977]: pgmap v383: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:37:40.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:40 smithi193 ceph-mon[52977]: pgmap v384: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:40.612 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:40 smithi031 ceph-mon[67456]: pgmap v384: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: pgmap v385: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 182 B/s rd, 0 op/s 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: pgmap v386: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:37:41.516 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.517 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:41 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: pgmap v385: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 182 B/s rd, 0 op/s 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: pgmap v386: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:41.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:41 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:43.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:43 smithi193 ceph-mon[52977]: Upgrade: Updating grafana.smithi031 2024-03-30T14:37:43.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:43 smithi193 ceph-mon[52977]: Deploying daemon grafana.smithi031 on smithi031 2024-03-30T14:37:43.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:43 smithi193 ceph-mon[52977]: pgmap v387: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:43.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:43 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:43.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:43 smithi031 ceph-mon[67456]: Upgrade: Updating grafana.smithi031 2024-03-30T14:37:43.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:43 smithi031 ceph-mon[67456]: Deploying daemon grafana.smithi031 on smithi031 2024-03-30T14:37:43.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:43 smithi031 ceph-mon[67456]: pgmap v387: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:43.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:43 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:45.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:45 smithi193 ceph-mon[52977]: pgmap v388: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:45.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:45 smithi031 ceph-mon[67456]: pgmap v388: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:47.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:47 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:47.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:47 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:47.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:47 smithi031 ceph-mon[67456]: pgmap v389: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:47.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:47 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:47.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:47 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:47.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:47 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:47.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:47 smithi193 ceph-mon[52977]: pgmap v389: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:47.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:47 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:49.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:49 smithi193 ceph-mon[52977]: pgmap v390: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:49.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:49 smithi031 ceph-mon[67456]: pgmap v390: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-03-30T14:37:51.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:51 smithi193 ceph-mon[52977]: pgmap v391: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:37:51.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:51 smithi031 ceph-mon[67456]: pgmap v391: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-30T14:37:53.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:53.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:52 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:53.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:52 smithi193 ceph-mon[52977]: pgmap v392: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:53.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:53.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:52 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:53.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:52 smithi031 ceph-mon[67456]: pgmap v392: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:54.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:54.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:53 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:54.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:54.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:53 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:55.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:54 smithi193 ceph-mon[52977]: pgmap v393: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:55.362 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:54 smithi031 ceph-mon[67456]: pgmap v393: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:37:57.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: pgmap v394: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: pgmap v395: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-30T14:37:57.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:57.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.842 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: pgmap v394: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:37:57.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:57.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:57.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: pgmap v395: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:37:57.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.864 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-30T14:37:57.865 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:57.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:58.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:58 smithi193 ceph-mon[52977]: Upgrade: Finalizing container_image settings 2024-03-30T14:37:58.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:58 smithi193 ceph-mon[52977]: Upgrade: Complete! 2024-03-30T14:37:58.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:58 smithi193 ceph-mon[52977]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-30T14:37:58.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:58 smithi193 ceph-mon[52977]: Cluster is now healthy 2024-03-30T14:37:58.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:58.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:58.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:58.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:58 smithi031 ceph-mon[67456]: Upgrade: Finalizing container_image settings 2024-03-30T14:37:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:58 smithi031 ceph-mon[67456]: Upgrade: Complete! 2024-03-30T14:37:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:58 smithi031 ceph-mon[67456]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-30T14:37:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:58 smithi031 ceph-mon[67456]: Cluster is now healthy 2024-03-30T14:37:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:37:58.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:37:59.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:37:59 smithi193 ceph-mon[52977]: pgmap v396: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:37:59.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:37:59 smithi031 ceph-mon[67456]: pgmap v396: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:01.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:01 smithi031 ceph-mon[67456]: pgmap v397: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:02.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:01 smithi193 ceph-mon[52977]: pgmap v397: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:03.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph orch ps' 2024-03-30T14:38:03.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:03 smithi031 ceph-mon[67456]: pgmap v398: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:03.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:03 smithi031 ceph-mon[67456]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:38:04.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:03 smithi193 ceph-mon[52977]: pgmap v398: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:04.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:03 smithi193 ceph-mon[52977]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:38:05.565 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:38:05.565 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (35s) 13s ago 21m 15.5M - 0.25.0 c8568f914cd2 d03e861104a9 2024-03-30T14:38:05.565 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (8m) 13s ago 21m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:38:05.565 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (8m) 66s ago 20m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:38:05.565 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (19s) 13s ago 21m 46.5M - 9.4.12 2bacad6d85d8 62d4bb2a1c4b 2024-03-30T14:38:05.565 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (2m) 13s ago 15m 35.8M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 5aea88780a6b 2024-03-30T14:38:05.565 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (114s) 66s ago 15m 12.5M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 8b29ef7503d9 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (11m) 13s ago 22m 523M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (10m) 66s ago 20m 453M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (9m) 13s ago 22m 64.0M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (9m) 66s ago 20m 49.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (100s) 13s ago 15m 58.2M - 5.7 b0c4478f1d83 c23318cf4683 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (94s) 66s ago 14m 59.6M - 5.7 b0c4478f1d83 30be1448d55c 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (76s) 13s ago 21m 7545k - 1.5.0 0da6a335fe13 0422feae07d9 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (69s) 66s ago 20m 5490k - 1.5.0 0da6a335fe13 1caae78a5321 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (7m) 13s ago 19m 434M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (6m) 13s ago 19m 400M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (6m) 13s ago 18m 375M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (5m) 13s ago 18m 416M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (4m) 66s ago 18m 344M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:38:05.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (4m) 66s ago 17m 288M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:38:05.567 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (3m) 66s ago 17m 283M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:38:05.567 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (2m) 66s ago 16m 285M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 62b9d172403e 2024-03-30T14:38:05.567 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (52s) 13s ago 20m 44.2M - 2.43.0 a07b618ecd1d c7e887b6e104 2024-03-30T14:38:05.862 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:05 smithi031 ceph-mon[67456]: pgmap v399: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:06.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:05 smithi193 ceph-mon[52977]: pgmap v399: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:06.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph versions' 2024-03-30T14:38:06.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:06 smithi031 ceph-mon[67456]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:38:07.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:06 smithi193 ceph-mon[52977]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:38:07.770 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:07 smithi031 ceph-mon[67456]: pgmap v400: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:08.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:07 smithi193 ceph-mon[52977]: pgmap v400: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-03-30T14:38:08.275 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 8 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:38:08.276 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 14 2024-03-30T14:38:08.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:38:08.277 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:38:08.810 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:08 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/532078862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:38:08.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-30T14:38:09.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:08 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/532078862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:38:09.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:09 smithi031 ceph-mon[67456]: pgmap v401: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:10.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:09 smithi193 ceph-mon[52977]: pgmap v401: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:10.578 INFO:teuthology.orchestra.run.smithi031.stdout:wait for servicemap items w/ changing names to refresh 2024-03-30T14:38:11.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'sleep 60' 2024-03-30T14:38:11.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:11 smithi031 ceph-mon[67456]: pgmap v402: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:11.888 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:11 smithi193 ceph-mon[52977]: pgmap v402: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:12.824 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:13.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:14.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:13 smithi193 ceph-mon[52977]: pgmap v403: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:14.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:13 smithi031 ceph-mon[67456]: pgmap v403: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:16.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:15 smithi193 ceph-mon[52977]: pgmap v404: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:16.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:15 smithi031 ceph-mon[67456]: pgmap v404: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:18.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:17 smithi193 ceph-mon[52977]: pgmap v405: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:18.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:17 smithi031 ceph-mon[67456]: pgmap v405: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:20.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:19 smithi193 ceph-mon[52977]: pgmap v406: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:20.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:19 smithi031 ceph-mon[67456]: pgmap v406: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:22.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:21 smithi193 ceph-mon[52977]: pgmap v407: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:22.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:21 smithi031 ceph-mon[67456]: pgmap v407: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:24.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:23 smithi193 ceph-mon[52977]: pgmap v408: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:24.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:23 smithi031 ceph-mon[67456]: pgmap v408: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:26.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:25 smithi193 ceph-mon[52977]: pgmap v409: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:26.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:25 smithi031 ceph-mon[67456]: pgmap v409: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:28.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:27 smithi193 ceph-mon[52977]: pgmap v410: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:28.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:27 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:28.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:27 smithi031 ceph-mon[67456]: pgmap v410: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:28.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:30.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:29 smithi193 ceph-mon[52977]: pgmap v411: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:30.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:29 smithi031 ceph-mon[67456]: pgmap v411: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:31 smithi193 ceph-mon[52977]: pgmap v412: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:32.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:31 smithi031 ceph-mon[67456]: pgmap v412: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:34.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:33 smithi193 ceph-mon[52977]: pgmap v413: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:34.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:33 smithi031 ceph-mon[67456]: pgmap v413: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:36.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:35 smithi193 ceph-mon[52977]: pgmap v414: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:36.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:35 smithi031 ceph-mon[67456]: pgmap v414: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:38.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:37 smithi193 ceph-mon[52977]: pgmap v415: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:38.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:37 smithi031 ceph-mon[67456]: pgmap v415: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:40.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:39 smithi193 ceph-mon[52977]: pgmap v416: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:40.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:39 smithi031 ceph-mon[67456]: pgmap v416: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:42.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:41 smithi193 ceph-mon[52977]: pgmap v417: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:42.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:41 smithi031 ceph-mon[67456]: pgmap v417: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:42.947 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:43.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:44.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:43 smithi193 ceph-mon[52977]: pgmap v418: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:44.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:43 smithi031 ceph-mon[67456]: pgmap v418: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:46.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:45 smithi193 ceph-mon[52977]: pgmap v419: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:46.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:45 smithi031 ceph-mon[67456]: pgmap v419: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:48.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:47 smithi193 ceph-mon[52977]: pgmap v420: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:48.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:47 smithi031 ceph-mon[67456]: pgmap v420: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:50.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:49 smithi193 ceph-mon[52977]: pgmap v421: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:50.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:49 smithi031 ceph-mon[67456]: pgmap v421: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:52.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:51 smithi193 ceph-mon[52977]: pgmap v422: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:52.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:51 smithi031 ceph-mon[67456]: pgmap v422: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:54.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:53 smithi193 ceph-mon[52977]: pgmap v423: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:54.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:53 smithi031 ceph-mon[67456]: pgmap v423: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:56.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:55 smithi193 ceph-mon[52977]: pgmap v424: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:56.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:55 smithi031 ceph-mon[67456]: pgmap v424: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:38:58.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:57 smithi193 ceph-mon[52977]: pgmap v425: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:58.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:38:58.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:58.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:38:58.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:57 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:38:58.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:57 smithi031 ceph-mon[67456]: pgmap v425: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:38:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:38:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:38:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:38:58.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:57 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:38:59.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:38:59.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:58 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:38:59.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:38:59.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:58 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' 2024-03-30T14:39:00.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:38:59 smithi193 ceph-mon[52977]: pgmap v426: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:00.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:38:59 smithi031 ceph-mon[67456]: pgmap v426: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:02.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:01 smithi193 ceph-mon[52977]: pgmap v427: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:02.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:01 smithi031 ceph-mon[67456]: pgmap v427: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:04.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:03 smithi193 ceph-mon[52977]: pgmap v428: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:04.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:03 smithi031 ceph-mon[67456]: pgmap v428: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:06.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:05 smithi193 ceph-mon[52977]: pgmap v429: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:06.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:05 smithi031 ceph-mon[67456]: pgmap v429: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:08.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:07 smithi193 ceph-mon[52977]: pgmap v430: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:08.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:07 smithi031 ceph-mon[67456]: pgmap v430: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:10.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:09 smithi193 ceph-mon[52977]: pgmap v431: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:10.112 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:09 smithi031 ceph-mon[67456]: pgmap v431: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:12.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:11 smithi193 ceph-mon[52977]: pgmap v432: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:12.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:11 smithi031 ceph-mon[67456]: pgmap v432: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:13.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:12 smithi193 ceph-mon[52977]: pgmap v433: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:13.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:12 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:39:13.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:12 smithi031 ceph-mon[67456]: pgmap v433: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:13.113 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:12 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:39:13.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph orch ps' 2024-03-30T14:39:15.724 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:alertmanager.smithi031 smithi031 *:9093,9094 running (105s) 84s ago 22m 15.5M - 0.25.0 c8568f914cd2 d03e861104a9 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi031 smithi031 running (9m) 84s ago 22m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4ae5dd0043d5 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:crash.smithi193 smithi193 running (9m) 2m ago 21m 6665k - 19.0.0-1967-g4e50bbcd b0c4478f1d83 f1b3baa5b5c0 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:grafana.smithi031 smithi031 *:3000 running (89s) 84s ago 22m 46.5M - 9.4.12 2bacad6d85d8 62d4bb2a1c4b 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi031.xsynqy smithi031 running (3m) 84s ago 16m 35.8M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 5aea88780a6b 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:mds.foofs.smithi193.shgxxs smithi193 running (3m) 2m ago 16m 12.5M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 8b29ef7503d9 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi031.lkrchi smithi031 *:8443,9283,8765 running (12m) 84s ago 23m 523M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 c3a4dba17e87 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.smithi193.ikzlnb smithi193 *:8443,9283,8765 running (11m) 2m ago 21m 453M - 19.0.0-1967-g4e50bbcd b0c4478f1d83 4c695c03753e 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi031 smithi031 running (11m) 84s ago 23m 64.0M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 6ab49835e68f 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:mon.smithi193 smithi193 running (10m) 2m ago 21m 49.2M 2048M 19.0.0-1967-g4e50bbcd b0c4478f1d83 d6d2414c2ac1 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.0.0.smithi031.juvmqh smithi031 *:2049 running (2m) 84s ago 16m 58.2M - 5.7 b0c4478f1d83 c23318cf4683 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:nfs.foo.1.0.smithi193.odcsji smithi193 *:2049 running (2m) 2m ago 16m 59.6M - 5.7 b0c4478f1d83 30be1448d55c 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi031 smithi031 *:9100 running (2m) 84s ago 22m 7545k - 1.5.0 0da6a335fe13 0422feae07d9 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:node-exporter.smithi193 smithi193 *:9100 running (2m) 2m ago 21m 5490k - 1.5.0 0da6a335fe13 1caae78a5321 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (8m) 84s ago 21m 434M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 c1ab3320f0df 2024-03-30T14:39:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (8m) 84s ago 20m 400M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 507a07329c98 2024-03-30T14:39:15.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi031 running (7m) 84s ago 20m 375M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 2b3faa63c708 2024-03-30T14:39:15.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (6m) 84s ago 19m 416M 1937M 19.0.0-1967-g4e50bbcd b0c4478f1d83 1de380dc2f05 2024-03-30T14:39:15.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi193 running (5m) 2m ago 19m 344M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 a93871da4517 2024-03-30T14:39:15.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi193 running (5m) 2m ago 18m 288M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 faf1b49d03f1 2024-03-30T14:39:15.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi193 running (4m) 2m ago 18m 283M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 8f73251ee641 2024-03-30T14:39:15.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi193 running (3m) 2m ago 18m 285M 2704M 19.0.0-1967-g4e50bbcd b0c4478f1d83 62b9d172403e 2024-03-30T14:39:15.726 INFO:teuthology.orchestra.run.smithi031.stdout:prometheus.smithi031 smithi031 *:9095 running (2m) 84s ago 21m 44.2M - 2.43.0 a07b618ecd1d c7e887b6e104 2024-03-30T14:39:16.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:15 smithi193 ceph-mon[52977]: pgmap v434: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:16.102 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:15 smithi031 ceph-mon[67456]: pgmap v434: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:16.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph orch upgrade status' 2024-03-30T14:39:17.957 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:17 smithi031 ceph-mon[67456]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:17.957 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:17 smithi031 ceph-mon[67456]: pgmap v435: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:18.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:17 smithi193 ceph-mon[52977]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:18.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:17 smithi193 ceph-mon[52977]: pgmap v435: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:18.397 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:39:18.397 INFO:teuthology.orchestra.run.smithi031.stdout: "target_image": null, 2024-03-30T14:39:18.398 INFO:teuthology.orchestra.run.smithi031.stdout: "in_progress": false, 2024-03-30T14:39:18.398 INFO:teuthology.orchestra.run.smithi031.stdout: "which": "", 2024-03-30T14:39:18.398 INFO:teuthology.orchestra.run.smithi031.stdout: "services_complete": [], 2024-03-30T14:39:18.398 INFO:teuthology.orchestra.run.smithi031.stdout: "progress": null, 2024-03-30T14:39:18.398 INFO:teuthology.orchestra.run.smithi031.stdout: "message": "", 2024-03-30T14:39:18.398 INFO:teuthology.orchestra.run.smithi031.stdout: "is_paused": false 2024-03-30T14:39:18.398 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:39:19.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph health detail' 2024-03-30T14:39:20.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:19 smithi031 ceph-mon[67456]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:20.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:19 smithi031 ceph-mon[67456]: pgmap v436: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:20.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:19 smithi193 ceph-mon[52977]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:20.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:19 smithi193 ceph-mon[52977]: pgmap v436: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:21.238 INFO:teuthology.orchestra.run.smithi031.stdout:HEALTH_OK 2024-03-30T14:39:21.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:21 smithi031 ceph-mon[67456]: pgmap v437: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:21.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:21 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2596880874' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:39:21.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph versions' 2024-03-30T14:39:22.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:21 smithi193 ceph-mon[52977]: pgmap v437: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:22.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:21 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2596880874' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T14:39:23.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:23 smithi031 ceph-mon[67456]: pgmap v438: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:23.919 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-30T14:39:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 8 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 2 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2024-03-30T14:39:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)": 14 2024-03-30T14:39:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:39:23.921 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-30T14:39:24.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:23 smithi193 ceph-mon[52977]: pgmap v438: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:24.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-30T14:39:24.838 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:24 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/2289729951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:39:25.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:24 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/2289729951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:39:25.767 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:25 smithi031 ceph-mon[67456]: pgmap v439: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:26.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:25 smithi193 ceph-mon[52977]: pgmap v439: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:26.716 INFO:teuthology.orchestra.run.smithi031.stdout:true 2024-03-30T14:39:27.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-30T14:39:27.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:27 smithi031 ceph-mon[67456]: pgmap v440: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:27.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:27 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1246612608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:39:27.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:27 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:39:28.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:27 smithi193 ceph-mon[52977]: pgmap v440: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:28.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:27 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1246612608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:39:28.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:27 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:39:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)" 2024-03-30T14:39:29.761 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:29 smithi031 ceph-mon[67456]: pgmap v441: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:29.761 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:29 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/3029615211' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:39:30.068 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-03-30T14:39:30.069 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-03-30T14:39:30.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-03-30T14:39:30.085 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:29 smithi193 ceph-mon[52977]: pgmap v441: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:30.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:29 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/3029615211' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T14:39:32.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:31 smithi193 ceph-mon[52977]: pgmap v442: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:32.098 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-30T14:39:32.098 INFO:teuthology.orchestra.run.smithi031.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-30T14:16:10.687846Z", "last_refresh": "2024-03-30T14:37:51.712311Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-30T14:16:04.765473Z", "last_refresh": "2024-03-30T14:36:59.371824Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-03-30T14:16:07.753420Z", "last_refresh": "2024-03-30T14:37:51.712466Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-30T14:22:35.908714Z", "last_refresh": "2024-03-30T14:36:59.372491Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-30T14:16:03.520237Z", "last_refresh": "2024-03-30T14:36:59.371927Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi031:172.21.15.31=smithi031", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-30T14:16:53.746502Z", "last_refresh": "2024-03-30T14:36:59.372018Z", "running": 2, "size": 2}}, {"events": ["2024-03-30T14:38:57.692173Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-03-30T14:22:56.880085Z", "last_refresh": "2024-03-30T14:36:59.372569Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-30T14:16:09.261132Z", "last_refresh": "2024-03-30T14:36:59.372102Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", "last_refresh": "2024-03-30T14:36:59.372181Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-30T14:16:06.520988Z", "last_refresh": "2024-03-30T14:37:51.712620Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-30T14:39:32.099 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:31 smithi031 ceph-mon[67456]: pgmap v442: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:32.748 INFO:tasks.cephadm:nfs.foo has 2/2 2024-03-30T14:39:32.748 DEBUG:teuthology.parallel:result is None 2024-03-30T14:39:32.748 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-30T14:39:32.758 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:39:32.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-03-30T14:39:32.787 INFO:teuthology.orchestra.run.smithi031.stderr:+ umount /mnt/foo 2024-03-30T14:39:32.799 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T14:39:32.808 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:39:32.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-03-30T14:39:33.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:33 smithi031 ceph-mon[67456]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:39:33.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:33 smithi031 ceph-mon[67456]: pgmap v443: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:34.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:33 smithi193 ceph-mon[52977]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:39:34.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:33 smithi193 ceph-mon[52977]: pgmap v443: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:34.908 INFO:teuthology.orchestra.run.smithi031.stdout: "foo" 2024-03-30T14:39:35.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs export ls foo --detailed' 2024-03-30T14:39:35.848 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:35 smithi031 ceph-mon[67456]: pgmap v444: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:36.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:35 smithi193 ceph-mon[52977]: pgmap v444: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:36.784 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:36 smithi031 ceph-mon[67456]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:37.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:36 smithi193 ceph-mon[52977]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:37.555 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "access_type": "RW", 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "clients": [], 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_id": "foo", 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "export_id": 1, 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "fsal": { 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "foofs", 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "CEPH", 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "user_id": "nfs.foo.1" 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-30T14:39:37.556 INFO:teuthology.orchestra.run.smithi031.stdout: "protocols": [ 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: "pseudo": "/fake", 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: "security_label": true, 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: "squash": "none", 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: "transports": [ 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: "TCP" 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-30T14:39:37.557 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-03-30T14:39:37.863 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:37 smithi031 ceph-mon[67456]: pgmap v445: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:38.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:37 smithi193 ceph-mon[52977]: pgmap v445: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-30T14:39:38.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'rados -p .nfs --all ls -' 2024-03-30T14:39:38.850 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:38 smithi031 ceph-mon[67456]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:39.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:38 smithi193 ceph-mon[52977]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:39:39.910 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:39 smithi031 ceph-mon[67456]: pgmap v446: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:39.910 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:39 smithi031 ceph-mon[67456]: mgrmap e44: smithi031.lkrchi(active, since 11m), standbys: smithi193.ikzlnb 2024-03-30T14:39:40.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:39 smithi193 ceph-mon[52977]: pgmap v446: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:40.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:39 smithi193 ceph-mon[52977]: mgrmap e44: smithi031.lkrchi(active, since 11m), standbys: smithi193.ikzlnb 2024-03-30T14:39:40.093 INFO:teuthology.orchestra.run.smithi031.stdout:foo rec-0000000000000004:nfs.foo.1 2024-03-30T14:39:40.093 INFO:teuthology.orchestra.run.smithi031.stdout:foo grace 2024-03-30T14:39:40.093 INFO:teuthology.orchestra.run.smithi031.stdout:foo rec-0000000000000004:nfs.foo.0 2024-03-30T14:39:40.093 INFO:teuthology.orchestra.run.smithi031.stdout:foo conf-nfs.foo 2024-03-30T14:39:40.093 INFO:teuthology.orchestra.run.smithi031.stdout:foo export-1 2024-03-30T14:39:40.741 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T14:39:40.751 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:39:40.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 -- bash -c 'set -ex 2024-03-30T14:39:40.751 DEBUG:teuthology.orchestra.run.smithi031:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-03-30T14:39:40.751 DEBUG:teuthology.orchestra.run.smithi031:> ' 2024-03-30T14:39:42.020 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:41 smithi031 ceph-mon[67456]: pgmap v447: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:42.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:41 smithi193 ceph-mon[52977]: pgmap v447: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-30T14:39:42.498 INFO:teuthology.orchestra.run.smithi031.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-03-30T14:39:42.767 INFO:teuthology.orchestra.run.smithi031.stderr:+ [[ 6 -gt 2 ]] 2024-03-30T14:39:42.947 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:42 smithi031 ceph-mon[67456]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:39:43.086 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:42 smithi193 ceph-mon[52977]: from='mgr.34104 172.21.15.31:0/436331462' entity='mgr.smithi031.lkrchi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:39:43.383 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-30T14:39:43.392 INFO:tasks.cephadm:Teardown begin 2024-03-30T14:39:43.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:39:43.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:39:43.455 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-30T14:39:43.455 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:39:43.475 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:39:43.513 INFO:tasks.cephadm:Stopping all daemons... 2024-03-30T14:39:43.513 INFO:tasks.cephadm.mon.smithi031:Stopping mon.smithi031... 2024-03-30T14:39:43.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031 2024-03-30T14:39:43.613 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:43 smithi031 systemd[1]: Stopping Ceph mon.smithi031 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:39:43.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:43 smithi193 ceph-mon[52977]: pgmap v448: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:39:43.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:43 smithi193 ceph-mon[52977]: from='client.? 172.21.15.31:0/1773172151' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-30T14:39:43.973 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:43 smithi031 ceph-mon[67456]: pgmap v448: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:39:43.973 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:43 smithi031 ceph-mon[67456]: from='client.? 172.21.15.31:0/1773172151' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-30T14:39:43.973 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[67450]: 2024-03-30T14:39:43.847+0000 7fa4251df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi031 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:39:43.973 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:43 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031[67450]: 2024-03-30T14:39:43.847+0000 7fa4251df700 -1 mon.smithi031@0(leader) e3 *** Got Signal Terminated *** 2024-03-30T14:39:44.363 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:44 smithi031 podman[113028]: 2024-03-30 14:39:44.071539337 +0000 UTC m=+0.340956678 container died 6ab49835e68fc948a4688d4b0102831d8dbe5751f91365a124236b4c49fc44f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:39:44.722 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:44 smithi031 podman[113028]: 2024-03-30 14:39:44.489539261 +0000 UTC m=+0.758956596 container cleanup 6ab49835e68fc948a4688d4b0102831d8dbe5751f91365a124236b4c49fc44f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:39:44.722 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 30 14:39:44 smithi031 bash[113028]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi031 2024-03-30T14:39:44.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi031.service' 2024-03-30T14:39:44.910 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:39:44.910 INFO:tasks.cephadm.mon.smithi031:Stopped mon.smithi031 2024-03-30T14:39:44.911 INFO:tasks.cephadm.mon.smithi193:Stopping mon.smithi193... 2024-03-30T14:39:44.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi193 2024-03-30T14:39:45.218 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:44 smithi193 systemd[1]: Stopping Ceph mon.smithi193 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:39:45.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193[52973]: 2024-03-30T14:39:45.218+0000 7fd6e2ad9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi193 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:39:45.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193[52973]: 2024-03-30T14:39:45.218+0000 7fd6e2ad9700 -1 mon.smithi193@1(peon) e3 *** Got Signal Terminated *** 2024-03-30T14:39:45.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:45 smithi193 podman[78214]: 2024-03-30 14:39:45.402993292 +0000 UTC m=+0.296881847 container died d6d2414c2ac116b4a095ff863a460a9ad4a40a1b4f79ad888157cd2e2dced53f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:39:46.035 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:45 smithi193 podman[78214]: 2024-03-30 14:39:45.827075529 +0000 UTC m=+0.720964083 container cleanup d6d2414c2ac116b4a095ff863a460a9ad4a40a1b4f79ad888157cd2e2dced53f (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-30T14:39:46.036 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Mar 30 14:39:45 smithi193 bash[78214]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-mon-smithi193 2024-03-30T14:39:46.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@mon.smithi193.service' 2024-03-30T14:39:46.304 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:39:46.304 INFO:tasks.cephadm.mon.smithi193:Stopped mon.smithi193 2024-03-30T14:39:46.304 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T14:39:46.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T14:39:46.609 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:46 smithi031 systemd[1]: Stopping Ceph osd.0 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:39:46.609 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:46 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[73454]: 2024-03-30T14:39:46.609+0000 7f4d918e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:39:46.609 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:46 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[73454]: 2024-03-30T14:39:46.609+0000 7f4d918e1700 -1 osd.0 156 *** Got signal Terminated *** 2024-03-30T14:39:46.609 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:46 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0[73454]: 2024-03-30T14:39:46.609+0000 7f4d918e1700 -1 osd.0 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:39:52.502 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:52 smithi031 podman[113160]: 2024-03-30 14:39:52.224075982 +0000 UTC m=+5.725728300 container died c1ab3320f0df33a978331cffae185e9e78ce1f0a49c8e14ef5e38d209003a7e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-30T14:39:52.863 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:52 smithi031 podman[113160]: 2024-03-30 14:39:52.5934883 +0000 UTC m=+6.095140616 container cleanup c1ab3320f0df33a978331cffae185e9e78ce1f0a49c8e14ef5e38d209003a7e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:39:52.863 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:52 smithi031 bash[113160]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T14:39:53.211 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:52 smithi031 podman[113179]: 2024-03-30 14:39:52.878522913 +0000 UTC m=+0.658399543 container remove c1ab3320f0df33a978331cffae185e9e78ce1f0a49c8e14ef5e38d209003a7e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0, RELEASE=squid-4e50bbc, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-30T14:39:53.555 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:53 smithi031 podman[113273]: 2024-03-30 14:39:53.212743988 +0000 UTC m=+0.112682785 container create 980da7c0fca7b32c31bd8030ce9b2246b70ec018ed81c78d36d34881f5dd04d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:39:53.555 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:53 smithi031 podman[113273]: 2024-03-30 14:39:53.120033949 +0000 UTC m=+0.019972767 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:39:53.556 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:53 smithi031 podman[113273]: 2024-03-30 14:39:53.457157419 +0000 UTC m=+0.357096230 container init 980da7c0fca7b32c31bd8030ce9b2246b70ec018ed81c78d36d34881f5dd04d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:39:53.556 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:53 smithi031 podman[113273]: 2024-03-30 14:39:53.460558351 +0000 UTC m=+0.360497147 container start 980da7c0fca7b32c31bd8030ce9b2246b70ec018ed81c78d36d34881f5dd04d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:39:53.556 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:53 smithi031 podman[113273]: 2024-03-30 14:39:53.556977641 +0000 UTC m=+0.456916432 container attach 980da7c0fca7b32c31bd8030ce9b2246b70ec018ed81c78d36d34881f5dd04d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326) 2024-03-30T14:39:53.863 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:53 smithi031 podman[113273]: 2024-03-30 14:39:53.769113912 +0000 UTC m=+0.669052710 container died 980da7c0fca7b32c31bd8030ce9b2246b70ec018ed81c78d36d34881f5dd04d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:39:54.311 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 30 14:39:54 smithi031 podman[113299]: 2024-03-30 14:39:54.310510225 +0000 UTC m=+0.535826508 container remove 980da7c0fca7b32c31bd8030ce9b2246b70ec018ed81c78d36d34881f5dd04d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-30T14:39:54.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T14:39:54.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:39:54.366 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T14:39:54.366 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-30T14:39:54.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.1 2024-03-30T14:39:54.613 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:39:54 smithi031 systemd[1]: Stopping Ceph osd.1 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:39:55.113 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:39:54 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[76063]: 2024-03-30T14:39:54.738+0000 7fb081bfa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:39:55.113 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:39:54 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[76063]: 2024-03-30T14:39:54.738+0000 7fb081bfa700 -1 osd.1 156 *** Got signal Terminated *** 2024-03-30T14:39:55.113 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:39:54 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1[76063]: 2024-03-30T14:39:54.738+0000 7fb081bfa700 -1 osd.1 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:40:00.363 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:00 smithi031 podman[113378]: 2024-03-30 14:40:00.098663113 +0000 UTC m=+5.492887162 container died 507a07329c9876699ffdf07436f535ad924630e905147d5213f989e374403ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-30T14:40:00.754 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:00 smithi031 podman[113378]: 2024-03-30 14:40:00.463059341 +0000 UTC m=+5.857283395 container cleanup 507a07329c9876699ffdf07436f535ad924630e905147d5213f989e374403ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-30T14:40:00.754 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:00 smithi031 bash[113378]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T14:40:01.099 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:00 smithi031 podman[113398]: 2024-03-30 14:40:00.755960111 +0000 UTC m=+0.651192894 container remove 507a07329c9876699ffdf07436f535ad924630e905147d5213f989e374403ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc) 2024-03-30T14:40:01.363 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:01 smithi031 podman[113490]: 2024-03-30 14:40:01.004180136 +0000 UTC m=+0.023568560 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:40:01.363 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:01 smithi031 podman[113490]: 2024-03-30 14:40:01.115117343 +0000 UTC m=+0.134505764 container create 9d1401b71cccbd6fedb5cebd773a107cf808126e2ab9ed3987f7c2fdad2cf890 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-30T14:40:01.683 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:01 smithi031 podman[113490]: 2024-03-30 14:40:01.375726423 +0000 UTC m=+0.395114845 container init 9d1401b71cccbd6fedb5cebd773a107cf808126e2ab9ed3987f7c2fdad2cf890 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T14:40:01.683 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:01 smithi031 podman[113490]: 2024-03-30 14:40:01.379042276 +0000 UTC m=+0.398430709 container start 9d1401b71cccbd6fedb5cebd773a107cf808126e2ab9ed3987f7c2fdad2cf890 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:40:01.683 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:01 smithi031 podman[113490]: 2024-03-30 14:40:01.459116188 +0000 UTC m=+0.478504626 container attach 9d1401b71cccbd6fedb5cebd773a107cf808126e2ab9ed3987f7c2fdad2cf890 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-03-30T14:40:02.027 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 30 14:40:01 smithi031 podman[113490]: 2024-03-30 14:40:01.684784495 +0000 UTC m=+0.704172934 container died 9d1401b71cccbd6fedb5cebd773a107cf808126e2ab9ed3987f7c2fdad2cf890 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-30T14:40:02.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-30T14:40:02.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:40:02.370 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-30T14:40:02.370 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T14:40:02.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T14:40:02.613 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:02 smithi031 systemd[1]: Stopping Ceph osd.2 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:40:03.113 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:02 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[78610]: 2024-03-30T14:40:02.740+0000 7f7b9a694700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:40:03.113 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:02 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[78610]: 2024-03-30T14:40:02.740+0000 7f7b9a694700 -1 osd.2 156 *** Got signal Terminated *** 2024-03-30T14:40:03.113 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:02 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2[78610]: 2024-03-30T14:40:02.740+0000 7f7b9a694700 -1 osd.2 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:40:08.539 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:08 smithi031 podman[113594]: 2024-03-30 14:40:08.151392955 +0000 UTC m=+5.523648506 container died 2b3faa63c7084ddb0b213e3cce5d920816f22731bbdc796c885456fee51de1b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:40:08.863 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:08 smithi031 podman[113594]: 2024-03-30 14:40:08.61477639 +0000 UTC m=+5.987031928 container cleanup 2b3faa63c7084ddb0b213e3cce5d920816f22731bbdc796c885456fee51de1b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-30T14:40:08.863 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:08 smithi031 bash[113594]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T14:40:09.249 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:08 smithi031 podman[113645]: 2024-03-30 14:40:08.907876439 +0000 UTC m=+0.756026219 container remove 2b3faa63c7084ddb0b213e3cce5d920816f22731bbdc796c885456fee51de1b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:40:09.510 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:09 smithi031 podman[113736]: 2024-03-30 14:40:09.250585732 +0000 UTC m=+0.109657930 container create 7d20f4f1e558993920df70fef957a46b65d51166887c25ce98df2bcaf97a9b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-30T14:40:09.510 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:09 smithi031 podman[113736]: 2024-03-30 14:40:09.158952435 +0000 UTC m=+0.018024660 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:40:09.510 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:09 smithi031 podman[113736]: 2024-03-30 14:40:09.511452457 +0000 UTC m=+0.370524673 container init 7d20f4f1e558993920df70fef957a46b65d51166887c25ce98df2bcaf97a9b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2) 2024-03-30T14:40:09.816 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:09 smithi031 podman[113736]: 2024-03-30 14:40:09.514893616 +0000 UTC m=+0.373965824 container start 7d20f4f1e558993920df70fef957a46b65d51166887c25ce98df2bcaf97a9b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-03-30T14:40:09.816 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:09 smithi031 podman[113736]: 2024-03-30 14:40:09.619677478 +0000 UTC m=+0.478749678 container attach 7d20f4f1e558993920df70fef957a46b65d51166887c25ce98df2bcaf97a9b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-30T14:40:10.113 INFO:journalctl@ceph.osd.2.smithi031.stdout:Mar 30 14:40:09 smithi031 podman[113736]: 2024-03-30 14:40:09.817588613 +0000 UTC m=+0.676660814 container died 7d20f4f1e558993920df70fef957a46b65d51166887c25ce98df2bcaf97a9b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-30T14:40:10.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T14:40:10.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:40:10.483 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T14:40:10.483 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-30T14:40:10.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.3 2024-03-30T14:40:10.833 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:10 smithi031 systemd[1]: Stopping Ceph osd.3 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:40:11.113 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:10 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:40:10.833+0000 7f722b301700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:40:11.113 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:10 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:40:10.833+0000 7f722b301700 -1 osd.3 156 *** Got signal Terminated *** 2024-03-30T14:40:11.113 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:10 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:40:10.833+0000 7f722b301700 -1 osd.3 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:40:12.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:12 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:12.449+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:13.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:13 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:13.409+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:13.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:13 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:13.364+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:13.863 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:13 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:40:13.397+0000 7f72238ed700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.605061+0000 front 2024-03-30T14:39:49.605194+0000 (oldest deadline 2024-03-30T14:40:13.105191+0000) 2024-03-30T14:40:14.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:14 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:14.406+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:14.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:14 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:14.388+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:14.863 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:14 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:40:14.438+0000 7f72238ed700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.605061+0000 front 2024-03-30T14:39:49.605194+0000 (oldest deadline 2024-03-30T14:40:13.105191+0000) 2024-03-30T14:40:15.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:15 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:15.453+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:15.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:15 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:15.415+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:15.863 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:15 smithi031 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3[81287]: 2024-03-30T14:40:15.426+0000 7f72238ed700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.605061+0000 front 2024-03-30T14:39:49.605194+0000 (oldest deadline 2024-03-30T14:40:13.105191+0000) 2024-03-30T14:40:16.591 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:16 smithi031 podman[113840]: 2024-03-30 14:40:16.306377814 +0000 UTC m=+5.585757579 container died 1de380dc2f05fa1d92969055d2ccf4a8f2e18919087ce5132241fad41a246b82 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:40:16.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:16 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:16.486+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:16.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:16 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:16.408+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:16.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:16 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:16.554+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:16.863 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:16 smithi031 podman[113840]: 2024-03-30 14:40:16.675072227 +0000 UTC m=+5.954451988 container cleanup 1de380dc2f05fa1d92969055d2ccf4a8f2e18919087ce5132241fad41a246b82 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD) 2024-03-30T14:40:16.863 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:16 smithi031 bash[113840]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T14:40:17.299 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:16 smithi031 podman[113856]: 2024-03-30 14:40:16.942905895 +0000 UTC m=+0.637588366 container remove 1de380dc2f05fa1d92969055d2ccf4a8f2e18919087ce5132241fad41a246b82 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-30T14:40:17.441 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:17 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:17.145+0000 7efc0e49e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:48.278586+0000 front 2024-03-30T14:39:48.278841+0000 (oldest deadline 2024-03-30T14:40:16.479120+0000) 2024-03-30T14:40:17.578 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:17 smithi031 podman[113952]: 2024-03-30 14:40:17.205545383 +0000 UTC m=+0.020497311 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:40:17.578 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:17 smithi031 podman[113952]: 2024-03-30 14:40:17.310381003 +0000 UTC m=+0.125332921 container create ad7a1c25978193251992c6c084f16163780cde59370c8ba341928780bd66b6b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-30T14:40:17.578 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:17 smithi031 podman[113952]: 2024-03-30 14:40:17.579434132 +0000 UTC m=+0.394386051 container init ad7a1c25978193251992c6c084f16163780cde59370c8ba341928780bd66b6b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:40:17.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:17 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:17.513+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:17.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:17 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:17.441+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:17.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:17 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:17.461+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:17.863 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:17 smithi031 podman[113952]: 2024-03-30 14:40:17.582501379 +0000 UTC m=+0.397453307 container start ad7a1c25978193251992c6c084f16163780cde59370c8ba341928780bd66b6b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-4e50bbc) 2024-03-30T14:40:17.863 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:17 smithi031 podman[113952]: 2024-03-30 14:40:17.67949012 +0000 UTC m=+0.494442062 container attach ad7a1c25978193251992c6c084f16163780cde59370c8ba341928780bd66b6b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-30T14:40:18.165 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 30 14:40:17 smithi031 podman[113952]: 2024-03-30 14:40:17.878059917 +0000 UTC m=+0.693011838 container died ad7a1c25978193251992c6c084f16163780cde59370c8ba341928780bd66b6b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-3-deactivate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T14:40:18.445 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:18 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:18.101+0000 7efc0e49e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:48.278586+0000 front 2024-03-30T14:39:48.278841+0000 (oldest deadline 2024-03-30T14:40:16.479120+0000) 2024-03-30T14:40:18.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-30T14:40:18.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:40:18.499 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-30T14:40:18.499 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T14:40:18.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T14:40:18.802 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:18 smithi193 systemd[1]: Stopping Ceph osd.4 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:40:18.803 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:18 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:18.444+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:18.803 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:18 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:18.543+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:18.803 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:18 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:18.445+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:19.086 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:18 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:18.802+0000 7efc15eb2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:40:19.086 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:18 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:18.802+0000 7efc15eb2700 -1 osd.4 156 *** Got signal Terminated *** 2024-03-30T14:40:19.086 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:18 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:18.802+0000 7efc15eb2700 -1 osd.4 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:40:19.409 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:19.100+0000 7efc0e49e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:48.278586+0000 front 2024-03-30T14:39:48.278841+0000 (oldest deadline 2024-03-30T14:40:16.479120+0000) 2024-03-30T14:40:19.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:19.571+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:19.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:19.409+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:19.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:19 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:19.482+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:20.459 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:20 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:20.096+0000 7efc0e49e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:48.278586+0000 front 2024-03-30T14:39:48.278841+0000 (oldest deadline 2024-03-30T14:40:16.479120+0000) 2024-03-30T14:40:20.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:20 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:20.493+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:20.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:20 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:20.585+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:20.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:20 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:20.459+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:21.448 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:21.132+0000 7efc0e49e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:48.278586+0000 front 2024-03-30T14:39:48.278841+0000 (oldest deadline 2024-03-30T14:40:16.479120+0000) 2024-03-30T14:40:21.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:21.448+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:21.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:21.448+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:21.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:21.448+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:21.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:21.448+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:21.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:21.540+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:21.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:21 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:21.540+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:22.440 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:22 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:22.161+0000 7efc0e49e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:48.278586+0000 front 2024-03-30T14:39:48.278841+0000 (oldest deadline 2024-03-30T14:40:16.479120+0000) 2024-03-30T14:40:22.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:22 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:22.461+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:22.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:22 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:22.461+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:22.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:22 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:22.574+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:22.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:22 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:22.574+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:22.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:22 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:22.440+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:22.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:22 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:22.440+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:23.437 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4[57959]: 2024-03-30T14:40:23.165+0000 7efc0e49e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:48.278586+0000 front 2024-03-30T14:39:48.278841+0000 (oldest deadline 2024-03-30T14:40:16.479120+0000) 2024-03-30T14:40:23.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:23.473+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:23.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:23.473+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:23.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:23.595+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:23.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:23.595+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:23.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:23.436+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:23.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:23 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:23.436+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:24.512 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:24 smithi193 podman[78368]: 2024-03-30 14:40:24.254328559 +0000 UTC m=+5.569220807 container died a93871da45177d16974095b2978153df53adeac58107fa70b4c3a1af2e96a48b (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T14:40:24.512 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:24 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:24.478+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:24.512 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:24 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:24.478+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:24.513 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:24 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:24.478+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:24.513 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:24 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:24.478+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:24.836 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:24 smithi193 podman[78368]: 2024-03-30 14:40:24.720667001 +0000 UTC m=+6.035559251 container cleanup a93871da45177d16974095b2978153df53adeac58107fa70b4c3a1af2e96a48b (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-30T14:40:24.836 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:24 smithi193 bash[78368]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T14:40:24.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:24 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:24.604+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:24.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:24 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:24.604+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:25.336 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:25 smithi193 podman[78417]: 2024-03-30 14:40:25.00577705 +0000 UTC m=+0.750648869 container remove a93871da45177d16974095b2978153df53adeac58107fa70b4c3a1af2e96a48b (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-30T14:40:25.674 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:25 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:25.484+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:25.674 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:25 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:25.484+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:25.674 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:25 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:25.572+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:25.674 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:25 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:25.572+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:25.675 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:25 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:25.465+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:25.675 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:25 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:25.465+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:25.675 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:25 smithi193 podman[78506]: 2024-03-30 14:40:25.339735885 +0000 UTC m=+0.113913988 container create a182f029230866f340f5d0731e62d22c1c1abeb890434aa5dff8ac1e42ec9946 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-30T14:40:25.675 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:25 smithi193 podman[78506]: 2024-03-30 14:40:25.244096586 +0000 UTC m=+0.018274712 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:40:25.675 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:25 smithi193 podman[78506]: 2024-03-30 14:40:25.584008957 +0000 UTC m=+0.358187074 container init a182f029230866f340f5d0731e62d22c1c1abeb890434aa5dff8ac1e42ec9946 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326) 2024-03-30T14:40:25.675 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:25 smithi193 podman[78506]: 2024-03-30 14:40:25.587242577 +0000 UTC m=+0.361420682 container start a182f029230866f340f5d0731e62d22c1c1abeb890434aa5dff8ac1e42ec9946 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:40:25.675 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:25 smithi193 podman[78506]: 2024-03-30 14:40:25.675673928 +0000 UTC m=+0.449852040 container attach a182f029230866f340f5d0731e62d22c1c1abeb890434aa5dff8ac1e42ec9946 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-30T14:40:26.086 INFO:journalctl@ceph.osd.4.smithi193.stdout:Mar 30 14:40:25 smithi193 podman[78506]: 2024-03-30 14:40:25.879132019 +0000 UTC m=+0.653310126 container died a182f029230866f340f5d0731e62d22c1c1abeb890434aa5dff8ac1e42ec9946 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True) 2024-03-30T14:40:26.494 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T14:40:26.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:40:26.537 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T14:40:26.537 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-30T14:40:26.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.5 2024-03-30T14:40:26.731 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:26.533+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:26.731 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:26.533+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:26.731 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:26 smithi193 systemd[1]: Stopping Ceph osd.5 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:40:26.731 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:26.617+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:26.731 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:26.617+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:26.731 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:26.446+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:26.731 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:26.446+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:27.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:26.898+0000 7f3a50c83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:40:27.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:26.898+0000 7f3a50c83700 -1 osd.5 156 *** Got signal Terminated *** 2024-03-30T14:40:27.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:26 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:26.898+0000 7f3a50c83700 -1 osd.5 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:40:27.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:27.448+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:27.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:27.448+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:27.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:27.581+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:27.837 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:27.581+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:27.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:27.620+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:27.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:27 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:27.620+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:28.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:28.409+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:28.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:28.409+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:28.569+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:28.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:28.569+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:28.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:28.622+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:28.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:28 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:28.622+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:29.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:29 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:29.425+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:29.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:29 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:29.425+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:29.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:29 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:29.425+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:29.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:29 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:29.610+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:29.837 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:29 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:29.610+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:29.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:29 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:29.639+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:29.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:29 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:29.639+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:30.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:30.440+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:30.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:30.440+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:30.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:30.440+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:30.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:30.652+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:30.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:30.652+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:30.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:30.657+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:30.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:30.657+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:30.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:30 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:30.657+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:31.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:31.650+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.060710+0000 front 2024-03-30T14:39:50.060870+0000 (oldest deadline 2024-03-30T14:40:12.360915+0000) 2024-03-30T14:40:31.836 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5[60247]: 2024-03-30T14:40:31.650+0000 7f3a4d277700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:57.661613+0000 front 2024-03-30T14:39:57.661722+0000 (oldest deadline 2024-03-30T14:40:21.161888+0000) 2024-03-30T14:40:31.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:31.650+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:31.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:31.650+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:31.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:31.650+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:31.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:31.482+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:31.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:31.482+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:31.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:31 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:31.482+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:32.633 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:32 smithi193 podman[78613]: 2024-03-30 14:40:32.348427074 +0000 UTC m=+5.574008537 container died faf1b49d03f17ccb79f7d0e9dd4d32fa119d61f612ee55b57def6cda5c777394 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:40:32.633 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:32 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:32.453+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:32.633 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:32 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:32.453+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:32.633 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:32 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:32.453+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:33.074 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:32 smithi193 podman[78613]: 2024-03-30 14:40:32.832612214 +0000 UTC m=+6.058193717 container cleanup faf1b49d03f17ccb79f7d0e9dd4d32fa119d61f612ee55b57def6cda5c777394 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-30T14:40:33.075 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:32 smithi193 bash[78613]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T14:40:33.075 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:32 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:32.633+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:33.075 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:32 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:32.633+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:33.075 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:32 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:32.633+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:33.336 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:33 smithi193 podman[78640]: 2024-03-30 14:40:33.075681728 +0000 UTC m=+0.722541884 container remove faf1b49d03f17ccb79f7d0e9dd4d32fa119d61f612ee55b57def6cda5c777394 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:40:33.649 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:33 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:33.480+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:33.650 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:33 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:33.480+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:33.650 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:33 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:33.480+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:33.650 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:33 smithi193 podman[78733]: 2024-03-30 14:40:33.359737668 +0000 UTC m=+0.096302957 container create 77f3f2780015c52f782669f5d00f87d2736ec5c234492412ce286d3b4ad6ff81 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-30T14:40:33.650 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:33 smithi193 podman[78733]: 2024-03-30 14:40:33.280907882 +0000 UTC m=+0.017473139 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:40:33.650 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:33 smithi193 podman[78733]: 2024-03-30 14:40:33.595508326 +0000 UTC m=+0.332073574 container init 77f3f2780015c52f782669f5d00f87d2736ec5c234492412ce286d3b4ad6ff81 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-03-30T14:40:33.650 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:33 smithi193 podman[78733]: 2024-03-30 14:40:33.598698469 +0000 UTC m=+0.335263716 container start 77f3f2780015c52f782669f5d00f87d2736ec5c234492412ce286d3b4ad6ff81 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-30T14:40:34.086 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:33 smithi193 podman[78733]: 2024-03-30 14:40:33.679575955 +0000 UTC m=+0.416141204 container attach 77f3f2780015c52f782669f5d00f87d2736ec5c234492412ce286d3b4ad6ff81 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-30T14:40:34.087 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 14:40:33 smithi193 podman[78733]: 2024-03-30 14:40:33.887745533 +0000 UTC m=+0.624310779 container died 77f3f2780015c52f782669f5d00f87d2736ec5c234492412ce286d3b4ad6ff81 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:40:34.087 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:33 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:33.650+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:34.087 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:33 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:33.650+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:34.087 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:33 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:33.650+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:34.460 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-30T14:40:34.505 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:40:34.505 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-30T14:40:34.505 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-30T14:40:34.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.6 2024-03-30T14:40:34.792 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:34 smithi193 systemd[1]: Stopping Ceph osd.6 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:40:34.792 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:34.666+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:34.792 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:34.666+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:34.792 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:34.666+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:34.792 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:34.793+0000 7fb2675be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:40:34.793 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:34.793+0000 7fb2675be700 -1 osd.6 156 *** Got signal Terminated *** 2024-03-30T14:40:34.793 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:34.793+0000 7fb2675be700 -1 osd.6 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:40:34.793 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:34.487+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:34.793 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:34.487+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:34.793 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:34 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:34.487+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:35.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:35 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:35.633+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:35.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:35 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:35.633+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:35.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:35 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:35.633+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:35.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:35 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:35.532+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:35.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:35 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:35.532+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:35.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:35 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:35.532+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:36.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:36.587+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:36.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:36.587+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:36.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:36.587+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:36.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:36.505+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:36.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:36.505+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:36.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:36 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:36.505+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:37.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:37 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:37.613+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:37.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:37 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:37.613+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:37.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:37 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:37.613+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:37.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:37 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:37.515+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:37.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:37 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:37.515+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:37.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:37 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:37.515+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:38.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:38 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:38.654+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:38.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:38 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:38.654+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:38.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:38 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:38.654+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:38.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:38 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:38.506+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:38.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:38 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:38.506+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:38.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:38 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:38.506+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:39.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:39.627+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:49.735049+0000 front 2024-03-30T14:39:49.735080+0000 (oldest deadline 2024-03-30T14:40:15.635274+0000) 2024-03-30T14:40:39.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:39.627+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:56.736572+0000 front 2024-03-30T14:39:56.736442+0000 (oldest deadline 2024-03-30T14:40:21.436462+0000) 2024-03-30T14:40:39.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:39.627+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:07.237797+0000 front 2024-03-30T14:40:07.237830+0000 (oldest deadline 2024-03-30T14:40:30.137881+0000) 2024-03-30T14:40:39.836 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6[62500]: 2024-03-30T14:40:39.627+0000 7fb25fbaa700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:10.138400+0000 front 2024-03-30T14:40:10.138368+0000 (oldest deadline 2024-03-30T14:40:39.538866+0000) 2024-03-30T14:40:39.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:39.467+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:39.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:39.467+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:39.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:39.467+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:39.837 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:39 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:39.467+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:40.570 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:40 smithi193 podman[78838]: 2024-03-30 14:40:40.309667992 +0000 UTC m=+5.619828469 container died 8f73251ee641f51400b7b0cf1e715c583663793379b444f0b0ccc62f8347c41c (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:40:40.570 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:40 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:40.437+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:40.570 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:40 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:40.437+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:40.570 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:40 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:40.437+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:40.570 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:40 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:40.437+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:40.837 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:40 smithi193 podman[78838]: 2024-03-30 14:40:40.676134454 +0000 UTC m=+5.986294919 container cleanup 8f73251ee641f51400b7b0cf1e715c583663793379b444f0b0ccc62f8347c41c (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:40:40.838 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:40 smithi193 bash[78838]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6 2024-03-30T14:40:41.229 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:40 smithi193 podman[78855]: 2024-03-30 14:40:40.918819245 +0000 UTC m=+0.613716701 container remove 8f73251ee641f51400b7b0cf1e715c583663793379b444f0b0ccc62f8347c41c (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-30T14:40:41.513 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:41 smithi193 podman[78949]: 2024-03-30 14:40:41.133469044 +0000 UTC m=+0.017916068 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:40:41.513 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:41 smithi193 podman[78949]: 2024-03-30 14:40:41.261709427 +0000 UTC m=+0.146156481 container create 87117b695a17bdc8024a27d22f3b28d5afbacc86a4cac7942596320b05bd2b69 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-30T14:40:41.513 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:41 smithi193 podman[78949]: 2024-03-30 14:40:41.514528051 +0000 UTC m=+0.398975072 container init 87117b695a17bdc8024a27d22f3b28d5afbacc86a4cac7942596320b05bd2b69 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-30T14:40:41.513 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:41.408+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:41.513 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:41.408+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:41.514 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:41.408+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:41.514 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:41 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:41.408+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:41.803 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:41 smithi193 podman[78949]: 2024-03-30 14:40:41.5178621 +0000 UTC m=+0.402309119 container start 87117b695a17bdc8024a27d22f3b28d5afbacc86a4cac7942596320b05bd2b69 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326) 2024-03-30T14:40:41.803 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:41 smithi193 podman[78949]: 2024-03-30 14:40:41.605686396 +0000 UTC m=+0.490133421 container attach 87117b695a17bdc8024a27d22f3b28d5afbacc86a4cac7942596320b05bd2b69 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc) 2024-03-30T14:40:42.086 INFO:journalctl@ceph.osd.6.smithi193.stdout:Mar 30 14:40:41 smithi193 podman[78949]: 2024-03-30 14:40:41.804741515 +0000 UTC m=+0.689188523 container died 87117b695a17bdc8024a27d22f3b28d5afbacc86a4cac7942596320b05bd2b69 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-30T14:40:42.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.6.service' 2024-03-30T14:40:42.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:40:42.459 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-30T14:40:42.459 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-30T14:40:42.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.7 2024-03-30T14:40:42.769 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:42.393+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:42.769 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:42.393+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:42.769 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:42.393+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:42.769 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:42.393+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:42.769 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 systemd[1]: Stopping Ceph osd.7 for dc60522e-ee9f-11ee-b647-cb9ed24678a4... 2024-03-30T14:40:43.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:42.769+0000 7fa6752f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:40:43.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:42.769+0000 7fa6752f8700 -1 osd.7 156 *** Got signal Terminated *** 2024-03-30T14:40:43.086 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:42 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:42.769+0000 7fa6752f8700 -1 osd.7 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:40:43.683 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:43 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:43.346+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:43.683 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:43 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:43.346+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:43.683 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:43 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:43.346+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:43.683 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:43 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:43.346+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:44.586 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:44 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:44.321+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:44.586 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:44 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:44.321+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:44.586 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:44 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:44.321+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:44.586 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:44 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:44.321+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:45.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:45.363+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:45.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:45.363+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:45.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:45.363+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:45.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:45.363+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:45.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:45 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:45.363+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.193:6804 osd.4 since back 2024-03-30T14:40:19.320959+0000 front 2024-03-30T14:40:19.321040+0000 (oldest deadline 2024-03-30T14:40:44.620881+0000) 2024-03-30T14:40:46.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:46 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:46.372+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:46.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:46 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:46.372+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:46.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:46 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:46.372+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:46.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:46 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:46.372+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:46.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:46 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:46.372+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.193:6804 osd.4 since back 2024-03-30T14:40:19.320959+0000 front 2024-03-30T14:40:19.321040+0000 (oldest deadline 2024-03-30T14:40:44.620881+0000) 2024-03-30T14:40:47.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:47 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:47.380+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-30T14:39:50.817241+0000 front 2024-03-30T14:39:50.817206+0000 (oldest deadline 2024-03-30T14:40:12.517159+0000) 2024-03-30T14:40:47.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:47 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:47.380+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-03-30T14:39:54.718569+0000 front 2024-03-30T14:39:54.718396+0000 (oldest deadline 2024-03-30T14:40:21.119002+0000) 2024-03-30T14:40:47.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:47 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:47.380+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-03-30T14:40:04.019841+0000 front 2024-03-30T14:40:04.019942+0000 (oldest deadline 2024-03-30T14:40:28.719809+0000) 2024-03-30T14:40:47.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:47 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:47.380+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-03-30T14:40:14.620710+0000 front 2024-03-30T14:40:14.620677+0000 (oldest deadline 2024-03-30T14:40:39.320604+0000) 2024-03-30T14:40:47.836 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:47 smithi193 ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7[64723]: 2024-03-30T14:40:47.380+0000 7fa6718ec700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.193:6804 osd.4 since back 2024-03-30T14:40:19.320959+0000 front 2024-03-30T14:40:19.321040+0000 (oldest deadline 2024-03-30T14:40:44.620881+0000) 2024-03-30T14:40:48.586 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:48 smithi193 podman[79056]: 2024-03-30 14:40:48.330485035 +0000 UTC m=+5.652333551 container died 62b9d172403e9641505f9cfb1a555092e28b7303e73075b187b6b4c270263a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-30T14:40:48.861 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:48 smithi193 podman[79056]: 2024-03-30 14:40:48.669306895 +0000 UTC m=+5.991155401 container cleanup 62b9d172403e9641505f9cfb1a555092e28b7303e73075b187b6b4c270263a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-30T14:40:48.861 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:48 smithi193 bash[79056]: ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7 2024-03-30T14:40:49.169 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:48 smithi193 podman[79118]: 2024-03-30 14:40:48.862146744 +0000 UTC m=+0.526354452 container remove 62b9d172403e9641505f9cfb1a555092e28b7303e73075b187b6b4c270263a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:40:49.422 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:49 smithi193 podman[79209]: 2024-03-30 14:40:49.171051978 +0000 UTC m=+0.114376922 container create d66d5b741955f168fa484421759a54c36c75b7b395263c75673459b2a8472cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-30T14:40:49.422 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:49 smithi193 podman[79209]: 2024-03-30 14:40:49.074874136 +0000 UTC m=+0.018199096 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:40:49.422 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:49 smithi193 podman[79209]: 2024-03-30 14:40:49.423314674 +0000 UTC m=+0.366639632 container init d66d5b741955f168fa484421759a54c36c75b7b395263c75673459b2a8472cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-30T14:40:49.712 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:49 smithi193 podman[79209]: 2024-03-30 14:40:49.426907251 +0000 UTC m=+0.370232205 container start d66d5b741955f168fa484421759a54c36c75b7b395263c75673459b2a8472cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T14:40:49.712 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:49 smithi193 podman[79209]: 2024-03-30 14:40:49.506688127 +0000 UTC m=+0.450013083 container attach d66d5b741955f168fa484421759a54c36c75b7b395263c75673459b2a8472cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-30T14:40:49.999 INFO:journalctl@ceph.osd.7.smithi193.stdout:Mar 30 14:40:49 smithi193 podman[79209]: 2024-03-30 14:40:49.714016976 +0000 UTC m=+0.657341928 container died d66d5b741955f168fa484421759a54c36c75b7b395263c75673459b2a8472cd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-30T14:40:50.282 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc60522e-ee9f-11ee-b647-cb9ed24678a4@osd.7.service' 2024-03-30T14:40:50.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:40:50.324 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-30T14:40:50.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T14:43:47.744 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T14:44:07.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:44:07.906 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:44:07.936 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-30T14:44:07.938 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164/remote/smithi031/crash 2024-03-30T14:44:07.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T14:44:07.985 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T14:44:07.986 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2024-03-30T14:44:07.987 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164/remote/smithi193/crash 2024-03-30T14:44:07.988 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T14:44:08.047 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T14:44:08.047 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-03-30T14:44:08.048 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-30T14:44:08.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-30T14:44:08.079 INFO:teuthology.orchestra.run.smithi031.stdout:2024-03-30T14:27:14.339071+0000 mon.smithi031 (mon.0) 873 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:44:08.080 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-30T14:44:08.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[SEC\]' /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-30T14:44:08.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]' /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-30T14:44:08.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[WRN\]' /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-30T14:44:08.292 INFO:teuthology.orchestra.run.smithi031.stdout:2024-03-30T14:27:14.339071+0000 mon.smithi031 (mon.0) 873 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-30T14:44:08.293 INFO:tasks.cephadm:Compressing logs... 2024-03-30T14:44:08.293 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T14:44:08.335 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T14:44:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mon.smithi031.log 2024-03-30T14:44:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:: No such file or directory 2024-03-30T14:44:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T14:44:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-03-30T14:44:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mon.smithi031.log: /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mgr.smithi031.lkrchi.log 2024-03-30T14:44:08.361 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mgr.smithi031.lkrchi.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi031.xsynqy.log 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi031.xsynqy.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr: 89.5% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi031.xsynqy.log.gz 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T14:44:08.367 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.2.log: 91.5% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:: No such file or directory 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mon.smithi193.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.6.log: /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mon.smithi193.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi193.shgxxs.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi193.shgxxs.log: /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr: 87.7% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi193.shgxxs.log.gz 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mgr.smithi193.ikzlnb.log 2024-03-30T14:44:08.370 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-30T14:44:08.371 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mgr.smithi193.ikzlnb.log: /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T14:44:08.371 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T14:44:08.371 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T14:44:08.371 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log: 91.6% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T14:44:08.371 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-30T14:44:08.372 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-volume.log: 87.6% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T14:44:08.372 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T14:44:08.372 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-30T14:44:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/cephadm.log: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T14:44:08.383 INFO:teuthology.orchestra.run.smithi193.stderr: 91.5% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T14:44:08.388 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/cephadm.log: 91.3% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T14:44:08.408 INFO:teuthology.orchestra.run.smithi031.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T14:44:08.434 INFO:teuthology.orchestra.run.smithi193.stderr: 89.7% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mgr.smithi193.ikzlnb.log.gz 2024-03-30T14:44:08.704 INFO:teuthology.orchestra.run.smithi193.stderr: 92.6% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mon.smithi193.log.gz 2024-03-30T14:44:08.781 INFO:teuthology.orchestra.run.smithi031.stderr: 88.4% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mgr.smithi031.lkrchi.log.gz 2024-03-30T14:44:10.413 INFO:teuthology.orchestra.run.smithi031.stderr: 91.9% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-mon.smithi031.log.gz 2024-03-30T14:44:21.535 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-03-30T14:44:23.146 INFO:teuthology.orchestra.run.smithi193.stderr: 93.9% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-03-30T14:44:24.955 INFO:teuthology.orchestra.run.smithi193.stderr: 94.2% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-03-30T14:44:27.226 INFO:teuthology.orchestra.run.smithi031.stderr: 94.1% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-03-30T14:44:27.627 INFO:teuthology.orchestra.run.smithi193.stderr: 94.2% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-03-30T14:44:27.821 INFO:teuthology.orchestra.run.smithi031.stderr: 94.1% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-03-30T14:44:28.094 INFO:teuthology.orchestra.run.smithi031.stderr: 94.2% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-03-30T14:44:28.096 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-30T14:44:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m19.749s 2024-03-30T14:44:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:user 1m11.050s 2024-03-30T14:44:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m2.610s 2024-03-30T14:44:32.242 INFO:teuthology.orchestra.run.smithi193.stderr: 94.1% -- replaced with /var/log/ceph/dc60522e-ee9f-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-03-30T14:44:32.243 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-30T14:44:32.243 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m23.894s 2024-03-30T14:44:32.243 INFO:teuthology.orchestra.run.smithi193.stderr:user 1m12.014s 2024-03-30T14:44:32.243 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m2.833s 2024-03-30T14:44:32.244 INFO:tasks.cephadm:Archiving logs... 2024-03-30T14:44:32.244 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164/remote/smithi031/log 2024-03-30T14:44:32.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T14:44:42.170 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164/remote/smithi193/log 2024-03-30T14:44:42.171 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T14:44:52.563 INFO:tasks.cephadm:Removing cluster... 2024-03-30T14:44:52.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 --force 2024-03-30T14:44:52.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc60522e-ee9f-11ee-b647-cb9ed24678a4 --force 2024-03-30T14:44:53.324 INFO:tasks.cephadm:Removing cephadm ... 2024-03-30T14:44:53.325 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T14:44:53.341 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T14:44:53.356 INFO:tasks.cephadm:Teardown complete 2024-03-30T14:44:53.356 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T14:44:53.367 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T14:44:53.367 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:44:53.383 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:44:53.395 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:44:53.400 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:44:53.400 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-03-30T14:44:53.400 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +11us[ +27us] +/- 73ms 2024-03-30T14:44:53.400 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -128us[ -112us] +/- 59ms 2024-03-30T14:44:53.400 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +200us[ +216us] +/- 81ms 2024-03-30T14:44:53.401 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:44:53.410 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:44:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:44:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-30T14:44:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +20us[ +45us] +/- 73ms 2024-03-30T14:44:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -134us[ -109us] +/- 59ms 2024-03-30T14:44:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +186us[ +211us] +/- 81ms 2024-03-30T14:44:53.414 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:44:53.415 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T14:44:53.424 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T14:44:53.425 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T14:44:53.435 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T14:44:53.527 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T14:44:53.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-30T14:44:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:44:53.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-30T14:44:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:44:53.666 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T14:44:53.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T14:44:53.687 INFO:teuthology.task.internal:Duration was 2065.780303 seconds 2024-03-30T14:44:53.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T14:44:53.697 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T14:44:53.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T14:44:53.701 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T14:44:53.742 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:44:53.748 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:44:54.155 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T14:44:54.155 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-03-30T14:44:54.156 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T14:44:54.216 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-03-30T14:44:54.216 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T14:44:54.251 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T14:44:54.252 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T14:44:54.258 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T14:44:54.320 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T14:44:54.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T14:44:54.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T14:45:08.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T14:45:08.305 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T14:45:08.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T14:45:08.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T14:45:08.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T14:45:08.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:45:08.381 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:45:08.403 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-03-30T14:45:08.436 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-03-30T14:45:08.462 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:45:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:45:08.484 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:45:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:45:08.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T14:45:08.530 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T14:45:08.531 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164/remote/smithi031 2024-03-30T14:45:08.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T14:45:08.601 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164/remote/smithi193 2024-03-30T14:45:08.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T14:45:08.650 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T14:45:08.650 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T14:45:08.653 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T14:45:08.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T14:45:08.714 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T14:45:08.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T14:45:08.726 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T14:45:08.726 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T14:45:08.728 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T14:45:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 14:45 /home/ubuntu/cephtest 2024-03-30T14:45:08.756 INFO:teuthology.orchestra.run.smithi193.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 14:45 /home/ubuntu/cephtest 2024-03-30T14:45:08.757 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T14:45:08.768 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T14:45:08.792 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T14:45:08.821 DEBUG:teuthology.nuke:shortname: smithi031 2024-03-30T14:45:08.821 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:45:08.847 DEBUG:teuthology.nuke:shortname: smithi193 2024-03-30T14:45:08.847 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:45:08.851 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:02:15.051617', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG89GpAQgzBB1UWS2gugrrqHdDANXgdwJyovs2hbGhOMFTjTrrVd0H68lzVS58IpenXBpmPyqrIWz5Y/lL0SbZw='} 2024-03-30T14:45:08.879 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630164', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:02:15.053156', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLiVOeIf5DBouL85wfOi0IeP7IQgbrFG+2MJnAbKpWXvILGURQlJANt8hpdFnadcB9cLlPE4hA5hiW4ZRKLT8/8='} 2024-03-30T14:45:08.909 INFO:teuthology.orchestra.console.smithi031:Power off 2024-03-30T14:45:08.909 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T14:45:08.942 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-03-30T14:45:08.943 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:08.961 INFO:teuthology.orchestra.console.smithi193:Power off 2024-03-30T14:45:08.961 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T14:45:08.973 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-30T14:45:08.987 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-03-30T14:45:08.987 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:09.015 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-30T14:45:12.974 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:13.016 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:13.086 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-30T14:45:13.127 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-30T14:45:17.090 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:17.127 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:17.202 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-30T14:45:17.239 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-30T14:45:21.206 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:21.240 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:45:21.317 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-03-30T14:45:21.350 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-03-30T14:45:21.419 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-03-30T14:45:21.451 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-03-30T14:45:21.556 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2065.7803032398224 failure_reason: '"2024-03-30T14:27:14.339071+0000 mon.smithi031 (mon.0) 873 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-30T14:45:21.556 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T14:45:21.649 INFO:teuthology.run:FAIL