2024-03-24T11:30:57.127 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T11:30:57.127 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932 --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932/orig.config.yaml 2024-03-24T11:30:57.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:30:57.267 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617932' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHla/g2BW76tkdDgkUHtnGhcCbXVqvUfdl8sdVsX4H8tQ9mLhsI+DMVJ83PchBBkU6L3/6wFg/YOksAlUPPue4U= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvGKTTtmLi+nEB3wCQRY0dsYqc9wyxZ4qLeJN2bz1MuKLHAsrAKaJuCChldvnIbWjUoTKB0/z/RtxeUxC+Sgkc= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-03-24T11:30:57.268 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T11:30:57.268 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T11:30:57.269 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T11:30:57.270 INFO:teuthology.task.internal:Checking packages... 2024-03-24T11:30:57.294 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T11:30:57.294 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:30:57.294 INFO:teuthology.packaging:ref: None 2024-03-24T11:30:57.294 INFO:teuthology.packaging:tag: None 2024-03-24T11:30:57.294 INFO:teuthology.packaging:branch: squid 2024-03-24T11:30:57.295 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:30:57.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T11:30:57.531 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T11:30:57.532 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T11:30:57.539 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T11:30:57.539 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T11:30:57.546 INFO:teuthology.task.internal:Saving configuration 2024-03-24T11:30:57.560 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T11:30:57.566 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:30:57.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:25:48.641389', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHla/g2BW76tkdDgkUHtnGhcCbXVqvUfdl8sdVsX4H8tQ9mLhsI+DMVJ83PchBBkU6L3/6wFg/YOksAlUPPue4U='} 2024-03-24T11:30:57.619 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:25:48.639381', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvGKTTtmLi+nEB3wCQRY0dsYqc9wyxZ4qLeJN2bz1MuKLHAsrAKaJuCChldvnIbWjUoTKB0/z/RtxeUxC+Sgkc='} 2024-03-24T11:30:57.619 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T11:30:57.625 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-24T11:30:57.625 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-24T11:30:57.625 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T11:30:57.708 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd0909e3f70>, signals=[15]) 2024-03-24T11:30:57.709 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T11:30:57.715 INFO:teuthology.task.internal:Opening connections... 2024-03-24T11:30:57.715 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:30:57.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:30:57.823 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-03-24T11:30:57.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:30:57.913 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T11:30:57.920 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-03-24T11:30:57.937 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-03-24T11:30:57.938 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-03-24T11:30:57.995 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-03-24T11:30:57.995 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-03-24T11:30:57.995 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-03-24T11:30:57.995 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:30:57.995 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-03-24T11:30:57.995 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:30:57.996 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:30:57.996 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-03-24T11:30:58.024 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-03-24T11:30:58.040 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-03-24T11:30:58.041 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:30:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:30:58.099 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:30:58.099 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:30:58.099 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-03-24T11:30:58.124 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T11:30:58.132 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T11:30:58.138 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T11:30:58.138 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:30:58.141 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:30:58.156 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T11:30:58.163 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T11:30:58.163 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:30:58.199 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:30:58.225 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T11:30:58.281 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T11:30:58.296 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T11:30:58.296 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T11:30:58.297 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:30:58.297 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:30:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.313 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-24T11:30:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.314 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-24T11:30:58.369 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:30:58.369 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-361.el9.x86_64 2024-03-24T11:30:58.370 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-03-24T11:30:58.372 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:30:58.372 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-361.el9.x86_64 2024-03-24T11:30:58.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-03-24T11:31:00.540 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T11:31:00.563 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T11:31:01.989 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-24T11:31:02.199 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-24T11:31:03.131 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T11:31:04.454 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-24T11:31:05.707 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-24T11:31:06.372 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-24T11:31:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-24T11:31:10.998 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.1 MB 00:00 2024-03-24T11:31:12.142 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-24T11:31:12.715 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-24T11:31:14.685 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 21 MB 00:02 2024-03-24T11:31:15.450 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-03-24T11:31:19.693 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-24T11:31:20.542 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-24T11:31:21.866 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:31:21.867 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:31:21.965 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-24T11:31:21.978 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-24T11:31:21.978 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-24T11:31:21.978 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-24T11:31:21.978 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-24T11:31:21.978 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:31:21.978 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-03-24T11:31:21.979 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:31:21.980 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 76 M 2024-03-24T11:31:21.980 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 123 M 2024-03-24T11:31:21.980 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-24T11:31:22.170 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 363 kB/s | 44 kB 00:00 2024-03-24T11:31:22.732 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:31:22.732 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:31:22.830 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-24T11:31:22.838 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-24T11:31:22.838 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-03-24T11:31:22.838 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-24T11:31:22.838 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-24T11:31:22.838 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:31:22.838 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-03-24T11:31:22.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:31:22.840 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 76 M 2024-03-24T11:31:22.840 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 123 M 2024-03-24T11:31:22.840 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-24T11:31:23.110 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 233 kB/s | 44 kB 00:00 2024-03-24T11:31:24.027 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-03-24T11:31:24.569 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-24T11:31:24.720 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 5.6 MB/s | 15 MB 00:02 2024-03-24T11:31:24.936 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 17 MB/s | 33 MB 00:02 2024-03-24T11:31:24.936 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:24.936 INFO:teuthology.orchestra.run.smithi032.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-24T11:31:25.630 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-24T11:31:26.058 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-24T11:31:26.059 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-24T11:31:26.670 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 6.1 MB/s | 27 MB 00:04 2024-03-24T11:31:27.113 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-24T11:31:27.113 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-24T11:31:27.753 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 5.9 MB/s | 33 MB 00:05 2024-03-24T11:31:27.754 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:27.754 INFO:teuthology.orchestra.run.smithi146.stdout:Total 13 MB/s | 76 MB 00:05 2024-03-24T11:31:28.422 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-24T11:31:28.796 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-24T11:31:28.796 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-24T11:31:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-24T11:31:29.799 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-24T11:31:29.799 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-24T11:31:30.384 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:31:30.478 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:31:32.266 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-24T11:31:32.871 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:31:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:31:33.347 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:31:33.482 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:31:35.677 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:31:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:31:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:31:38.601 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:31:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:31:40.562 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:31:43.695 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:31:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:14.698 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:14.698 INFO:teuthology.orchestra.run.smithi032.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:32:14.698 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:32:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:14.712 INFO:teuthology.orchestra.run.smithi146.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:32:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:32:14.745 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:15.113 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:15.113 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:32:15.113 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:32:15.113 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:32:15.162 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:15.162 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:32:15.162 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:32:15.163 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:32:15.557 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:15.557 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:32:15.557 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-24T11:32:15.557 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.557 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.557 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.558 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.558 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:32:15.558 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-24T11:32:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:32:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:32:15.581 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-24T11:32:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:32:15.582 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-24T11:32:15.899 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:32:15.951 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:32:15.995 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:32:15.996 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:32:15.996 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:32:15.996 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:32:15.996 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:32:15.996 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-24T11:32:15.996 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:32:15.996 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:32:16.064 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:32:16.064 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:32:16.064 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:32:16.064 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:32:16.065 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:32:16.065 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-24T11:32:16.065 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:32:16.065 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:32:16.078 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:32:16.079 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:32:16.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-03-24T11:32:16.116 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-03-24T11:32:16.116 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-03-24T11:32:16.116 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-03-24T11:32:16.116 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 68.el9 2024-03-24T11:32:16.116 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-03-24T11:32:16.116 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8264254 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:32:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:32:16.118 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:32:16.118 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:32:16.118 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-03-24T11:32:16.118 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:32:16.118 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:32:16.119 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:32:16.119 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:32:16.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:32:16.144 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:32:16.145 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:32:16.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-03-24T11:32:16.178 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-03-24T11:32:16.179 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:32:16.180 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:32:16.181 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:32:16.181 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:32:16.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:32:16.776 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-03-24T11:32:16.824 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-03-24T11:32:18.184 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:32:18.216 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-03-24T11:32:18.218 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-24T11:32:18.233 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.S8fc0u2olC 2024-03-24T11:32:18.233 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.S8fc0u2olC 2024-03-24T11:32:18.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.S8fc0u2olC 2024-03-24T11:32:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:32:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-03-24T11:32:18.351 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-24T11:32:18.366 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.m3ptcwdIYh 2024-03-24T11:32:18.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.m3ptcwdIYh 2024-03-24T11:32:18.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.m3ptcwdIYh 2024-03-24T11:32:18.442 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.S8fc0u2olC is 6KB 2024-03-24T11:32:18.461 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.S8fc0u2olC 2024-03-24T11:32:18.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:32:18.543 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:32:18.544 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:32:18.544 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:32:18.544 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:32:18.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:32:18.576 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.m3ptcwdIYh is 6KB 2024-03-24T11:32:18.624 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.m3ptcwdIYh 2024-03-24T11:32:18.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:32:18.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-03-24T11:32:18.707 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:32:18.707 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:32:18.707 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:32:18.707 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:32:18.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:32:18.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-03-24T11:32:48.730 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-24T11:32:48.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-24T11:32:48.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:32:48.870 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-24T11:32:48.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-24T11:32:48.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:33:23.359 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-24T11:33:32.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-24T11:33:32.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:33:48.890 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:33:50.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-24T11:33:57.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-24T11:33:57.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:33:58.185 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-24T11:33:58.563 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-24T11:33:58.563 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:33:58.563 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-24T11:33:58.578 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:33:58.578 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:33:58.579 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:33:58.579 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:33:59.579 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-24T11:33:59.579 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:33:59.665 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-03-24T11:33:59.701 DEBUG:teuthology.parallel:result is None 2024-03-24T11:34:02.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-24T11:34:02.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:34:05.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-24T11:34:20.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-24T11:34:20.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:34:21.215 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-24T11:34:21.648 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-24T11:34:21.648 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:34:21.648 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-24T11:34:21.665 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:34:21.666 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:34:21.666 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:34:21.666 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:34:22.667 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-24T11:34:22.667 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:34:22.735 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-03-24T11:34:22.775 DEBUG:teuthology.parallel:result is None 2024-03-24T11:34:22.775 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:34:22.782 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:34:22.782 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:34:22.785 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:34:22.802 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:34:22.809 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:34:22.817 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:34:22.817 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:34:22.843 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:34:22.870 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:34:22.878 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:34:22.879 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:34:22.909 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:34:22.942 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:34:22.947 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:34:22.956 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:34:22.962 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:34:22.969 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:34:22.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:34:23.005 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:34:23.035 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:34:23.044 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:34:23.044 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:34:23.074 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:34:23.091 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:34:23.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:34:23.226 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:34:23.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:34:23.351 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:34:23.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:34:23.418 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:34:23.464 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:34:23.520 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:34:23.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:34:23.648 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-24T11:34:23.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:34:23.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-03-24T11:34:23.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-03-24T11:34:23.765 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:34:23.806 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:34:24.238 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:34:24.246 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:34:24.246 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:34:24.264 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:34:24.272 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T11:34:24.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-03-24T11:34:24.333 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-03-24T11:34:24.335 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-03-24T11:34:24.397 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-03-24T11:34:24.399 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:34:24.399 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-03-24T11:34:24.431 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-03-24T11:34:24.432 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-03-24T11:34:24.457 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-03-24T11:34:24.458 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:34:24.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:34:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:34:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.529 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:34:24.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:34:24.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:34:24.610 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:34:24.620 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T11:34:24.624 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T11:34:24.746 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:34:24.762 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:34:24.764 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:38:36.849 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-03-24T11:38:36.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-24T11:38:36.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:36.944 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-24T11:38:37.030 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-24T11:38:37.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-24T11:38:37.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:37.120 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-24T11:38:37.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-24T11:38:37.205 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:38:37.218 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:38:37.218 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:38:37.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:38:37.221 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:38:37.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:38:37.251 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:38:37.268 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:38:37.277 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:38:37.294 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:38:37.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-03-24T11:38:37.311 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-03-24T11:38:37.323 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-03-24T11:38:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:38:37.336 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-03-24T11:38:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:38:37.353 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:38:37.370 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:38:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:38:37.401 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:38:37.401 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-03-24T11:38:37.401 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.401 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.401 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.401 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.419 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:38:37.423 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:38:37.424 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-24T11:38:37.424 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.424 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.425 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.425 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:38:37.425 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:38:37.520 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:38:37.521 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-24T11:38:37.521 INFO:tasks.cephadm:Cluster fsid is 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:38:37.521 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:38:37.521 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-24T11:38:37.521 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi032': '172.21.15.32', 'mon.smithi146': '172.21.15.146'} 2024-03-24T11:38:37.521 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:38:37.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-03-24T11:38:37.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-03-24T11:38:37.578 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-24T11:38:37.578 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:38:37.811 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 24 11:38 /home/ubuntu/cephtest/cephadm 2024-03-24T11:38:37.811 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:38:38.053 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 24 11:38 /home/ubuntu/cephtest/cephadm 2024-03-24T11:38:38.053 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:38:38.069 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:38:38.092 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-24T11:38:38.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-24T11:38:38.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-24T11:38:38.965 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:38:39.003 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:38:55.564 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-24T11:38:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-24T11:38:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-24T11:38:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2024-03-24T11:38:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-24T11:38:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-24T11:38:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-24T11:38:55.565 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-24T11:38:58.783 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:38:58.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-03-24T11:38:58.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-03-24T11:38:58.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-03-24T11:38:58.929 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-03-24T11:38:58.973 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:38:58.975 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:38:58.975 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:38:58.976 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:38:58.976 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:38:58.976 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:38:58.976 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:38:58.976 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:38:58.976 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T11:38:58.977 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:38:58.978 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:38:58.993 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0ddfea52-e9d3-11ee-95cd-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-24T11:38:58.994 DEBUG:teuthology.orchestra.run.smithi032:mon.smithi032> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service 2024-03-24T11:38:59.036 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:38:59.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.32 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:38:59.236 INFO:teuthology.orchestra.run.smithi032.stderr:-------------------------------------------------------------------------------- 2024-03-24T11:38:59.236 INFO:teuthology.orchestra.run.smithi032.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '0ddfea52-e9d3-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.32', '--skip-admin-label'] 2024-03-24T11:38:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: 4.9.3-dev 2024-03-24T11:38:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:Verifying podman|docker is present... 2024-03-24T11:38:59.282 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: 4.9.3-dev 2024-03-24T11:38:59.284 INFO:teuthology.orchestra.run.smithi032.stderr:Verifying lvm2 is present... 2024-03-24T11:38:59.284 INFO:teuthology.orchestra.run.smithi032.stderr:Verifying time synchronization is in place... 2024-03-24T11:38:59.291 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:38:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: inactive 2024-03-24T11:38:59.303 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: enabled 2024-03-24T11:38:59.309 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: active 2024-03-24T11:38:59.310 INFO:teuthology.orchestra.run.smithi032.stderr:Unit chronyd.service is enabled and running 2024-03-24T11:38:59.310 INFO:teuthology.orchestra.run.smithi032.stderr:Repeating the final host check... 2024-03-24T11:38:59.330 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: 4.9.3-dev 2024-03-24T11:38:59.332 INFO:teuthology.orchestra.run.smithi032.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:38:59.332 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl is present 2024-03-24T11:38:59.332 INFO:teuthology.orchestra.run.smithi032.stderr:lvcreate is present 2024-03-24T11:38:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:38:59.344 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: inactive 2024-03-24T11:38:59.350 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: enabled 2024-03-24T11:38:59.356 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: active 2024-03-24T11:38:59.357 INFO:teuthology.orchestra.run.smithi032.stderr:Unit chronyd.service is enabled and running 2024-03-24T11:38:59.357 INFO:teuthology.orchestra.run.smithi032.stderr:Host looks OK 2024-03-24T11:38:59.357 INFO:teuthology.orchestra.run.smithi032.stderr:Cluster fsid: 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:38:59.357 INFO:teuthology.orchestra.run.smithi032.stderr:Acquiring lock 140338563563040 on /run/cephadm/0ddfea52-e9d3-11ee-95cd-87774f69a715.lock 2024-03-24T11:38:59.357 INFO:teuthology.orchestra.run.smithi032.stderr:Lock 140338563563040 acquired on /run/cephadm/0ddfea52-e9d3-11ee-95cd-87774f69a715.lock 2024-03-24T11:38:59.358 INFO:teuthology.orchestra.run.smithi032.stderr:Verifying IP 172.21.15.32 port 3300 ... 2024-03-24T11:38:59.358 INFO:teuthology.orchestra.run.smithi032.stderr:Verifying IP 172.21.15.32 port 6789 ... 2024-03-24T11:38:59.359 INFO:teuthology.orchestra.run.smithi032.stderr:Base mon IP is 172.21.15.32, final addrv is [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-03-24T11:38:59.361 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.32 metric 100 2024-03-24T11:38:59.361 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 metric 100 2024-03-24T11:38:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:38:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T11:38:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T11:38:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:38:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-24T11:38:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-24T11:38:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T11:38:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-24T11:38:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-24T11:38:59.367 INFO:teuthology.orchestra.run.smithi032.stderr:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-03-24T11:38:59.367 INFO:teuthology.orchestra.run.smithi032.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:38:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:38:59.472 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:38:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Getting image source signatures 2024-03-24T11:38:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-24T11:38:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-24T11:38:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-24T11:38:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-24T11:38:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-24T11:38:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-24T11:38:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: Writing manifest to image destination 2024-03-24T11:39:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-24T11:39:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-24T11:39:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-24T11:39:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:Extracting ceph user uid/gid from container image... 2024-03-24T11:39:01.287 INFO:teuthology.orchestra.run.smithi032.stderr:stat: 167 167 2024-03-24T11:39:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:Creating initial keys... 2024-03-24T11:39:02.168 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph-authtool: AQDWEABm3J6DBBAAhJSy1ec+mi/TTmyfR0EOkQ== 2024-03-24T11:39:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph-authtool: AQDWEABm1Rl4OxAAaw45TPTeANtEATlPFfO4Kg== 2024-03-24T11:39:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph-authtool: AQDXEABmJtZcMRAAQALP4owt+Hr0H9p8qI8eOA== 2024-03-24T11:39:04.418 INFO:teuthology.orchestra.run.smithi032.stderr:Creating initial monmap... 2024-03-24T11:39:04.833 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:39:04.834 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-24T11:39:04.834 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:04.834 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:39:05.340 INFO:teuthology.orchestra.run.smithi032.stderr:monmaptool for smithi032 [v2:172.21.15.32:3300,v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:39:05.340 INFO:teuthology.orchestra.run.smithi032.stderr:setting min_mon_release = octopus 2024-03-24T11:39:05.340 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/monmaptool: set fsid to 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:05.341 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:39:05.341 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:39:05.341 INFO:teuthology.orchestra.run.smithi032.stderr:Creating mon... 2024-03-24T11:39:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:create mon.smithi032 on 2024-03-24T11:39:06.449 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:39:06.592 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715.target. 2024-03-24T11:39:06.592 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715.target. 2024-03-24T11:39:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Failed to reset failed state of unit ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service: Unit ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service not loaded. 2024-03-24T11:39:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715.target.wants/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service → /etc/systemd/system/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@.service. 2024-03-24T11:39:07.526 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:07 smithi032 ceph-mon[24444]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-03-24T11:39:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: disabled 2024-03-24T11:39:07.573 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: inactive 2024-03-24T11:39:07.575 INFO:teuthology.orchestra.run.smithi032.stderr:firewalld.service is not enabled 2024-03-24T11:39:07.575 INFO:teuthology.orchestra.run.smithi032.stderr:Not possible to enable service . firewalld.service is not available 2024-03-24T11:39:07.575 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for mon to start... 2024-03-24T11:39:07.575 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for mon... 2024-03-24T11:39:07.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:07 smithi032 bash[24425]: 1623925d60191a0cef64f142cbc81497595b90180cc738c20ada7dea1f3f990c 2024-03-24T11:39:07.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:07 smithi032 systemd[1]: Started Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:39:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: cluster: 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: id: 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: services: 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi032 (age 0.550986s) 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: data: 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:39:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: pgs: 2024-03-24T11:39:08.077 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:08.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:08 smithi032 ceph-mon[24444]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-03-24T11:39:08.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:08 smithi032 ceph-mon[24444]: monmap e1: 1 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} 2024-03-24T11:39:08.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:08 smithi032 ceph-mon[24444]: fsmap 2024-03-24T11:39:08.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:08 smithi032 ceph-mon[24444]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:39:08.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:08 smithi032 ceph-mon[24444]: mgrmap e1: no daemons active 2024-03-24T11:39:08.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:08 smithi032 ceph-mon[24444]: from='client.? 172.21.15.32:0/451069614' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T11:39:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:mon is available 2024-03-24T11:39:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:Assimilating anything we can from ceph.conf... 2024-03-24T11:39:09.182 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:09.182 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: [global] 2024-03-24T11:39:09.182 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: fsid = 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:09.182 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: [mgr] 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: [osd] 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-24T11:39:09.183 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-24T11:39:09.709 INFO:teuthology.orchestra.run.smithi032.stderr:Generating new minimal ceph.conf... 2024-03-24T11:39:09.816 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:09 smithi032 ceph-mon[24444]: from='client.? 172.21.15.32:0/903170486' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:39:09.816 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:09 smithi032 ceph-mon[24444]: from='client.? 172.21.15.32:0/903170486' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:39:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:Restarting the monitor... 2024-03-24T11:39:10.746 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:10 smithi032 ceph-mon[24444]: from='client.? 172.21.15.32:0/237559099' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:10.746 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:10 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:39:11.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:10 smithi032 bash[24663]: Error: no container with name or ID "ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon.smithi032" found: no such container 2024-03-24T11:39:11.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:10 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[24440]: 2024-03-24T11:39:10.833+0000 7f8a4b5c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:39:11.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:10 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[24440]: 2024-03-24T11:39:10.833+0000 7f8a4b5c3700 -1 mon.smithi032@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:39:11.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:10 smithi032 podman[24675]: 2024-03-24 11:39:10.914775718 +0000 UTC m=+0.160312597 container died 1623925d60191a0cef64f142cbc81497595b90180cc738c20ada7dea1f3f990c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, version=8, ceph=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-24T11:39:11.424 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 podman[24675]: 2024-03-24 11:39:11.207159065 +0000 UTC m=+0.452695951 container cleanup 1623925d60191a0cef64f142cbc81497595b90180cc738c20ada7dea1f3f990c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, distribution-scope=public, io.buildah.version=1.19.8, version=8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-24T11:39:11.425 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 bash[24675]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032 2024-03-24T11:39:11.425 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 bash[24700]: Error: no container with name or ID "ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon.smithi032" found: no such container 2024-03-24T11:39:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 podman[24689]: 2024-03-24 11:39:11.424625061 +0000 UTC m=+0.508576145 container remove 1623925d60191a0cef64f142cbc81497595b90180cc738c20ada7dea1f3f990c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, GIT_BRANCH=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.buildah.version=1.19.8, vcs-type=git, release=754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, architecture=x86_64, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T11:39:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service: Deactivated successfully. 2024-03-24T11:39:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 systemd[1]: Stopped Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:39:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 systemd[1]: Starting Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:39:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 podman[24774]: 2024-03-24 11:39:11.650320108 +0000 UTC m=+0.104193849 container create 4da956fd8f670ef3a51012f8e9b434728a4c6a98172da7a7084ee96c2d500031 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, ceph=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, distribution-scope=public, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-24T11:39:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 podman[24774]: 2024-03-24 11:39:11.564799363 +0000 UTC m=+0.018673075 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-24T11:39:11.972 INFO:teuthology.orchestra.run.smithi032.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-24T11:39:12.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 podman[24774]: 2024-03-24 11:39:11.8862213 +0000 UTC m=+0.340095009 container init 4da956fd8f670ef3a51012f8e9b434728a4c6a98172da7a7084ee96c2d500031 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, maintainer=Guillaume Abrioux , architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T11:39:12.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 podman[24774]: 2024-03-24 11:39:11.889241455 +0000 UTC m=+0.343115166 container start 4da956fd8f670ef3a51012f8e9b434728a4c6a98172da7a7084ee96c2d500031 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, architecture=x86_64, RELEASE=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_BRANCH=HEAD, distribution-scope=public, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T11:39:12.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:39:12.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-24T11:39:12.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: pidfile_write: ignore empty --pid-file 2024-03-24T11:39:12.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: load: jerasure load: lrc 2024-03-24T11:39:12.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: RocksDB version: 6.15.5 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Compile date Apr 18 2022 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: DB SUMMARY 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: DB Session ID: 18FABI4C023OOBG3IYRZ 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-24T11:39:12.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi032/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi032/store.db: 000010.log size: 82699 ; 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.env: 0x55b52bcc6860 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.fs: Posix File System 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.info_log: 0x55b52daa1fa0 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.statistics: (nil) 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.use_fsync: 0 2024-03-24T11:39:12.098 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.db_log_dir: 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi032/store.db 2024-03-24T11:39:12.099 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.write_buffer_manager: 0x55b52db92330 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-24T11:39:12.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.unordered_write: 0 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.row_cache: None 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.wal_filter: None 2024-03-24T11:39:12.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.preserve_deletes: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:39:12.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:39:12.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_open_files: -1 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Compression algorithms supported: 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kZSTD supported: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:39:12.104 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000009 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.merge_operator: 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_filter: None 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b52da6dd00) 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:39:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_type: 0 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_index_type: 0 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_shortening: 1 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: hash_index_allow_collision: 1 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: checksum: 1 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: no_block_cache: 0 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache: 0x55b52dad8170 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_options: 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: capacity : 536870912 2024-03-24T11:39:12.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_shard_bits : 4 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: strict_capacity_limit : 0 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_compressed: (nil) 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: persistent_cache: (nil) 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size: 4096 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size_deviation: 10 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_restart_interval: 16 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_block_restart_interval: 1 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_block_size: 4096 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: partition_filters: 0 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: use_delta_encoding: 1 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: whole_key_filtering: 1 2024-03-24T11:39:12.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: verify_compression: 0 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: format_version: 4 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_index_compression: 1 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_align: 0 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression: NoCompression 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.num_levels: 7 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:39:12.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:39:12.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:39:12.110 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:39:12.110 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:39:12.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.arena_block_size: 4194304 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:39:12.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.ttl: 2592000 2024-03-24T11:39:12.113 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280351915267, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280351915987, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79381, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77645, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71954, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711280351, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "fca7b695-5727-4e2a-9c91-a2f9ba2d792a", "db_session_id": "18FABI4C023OOBG3IYRZ"}} 2024-03-24T11:39:12.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280351916336, "job": 1, "event": "recovery_finished"} 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi032/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55b52dabea80 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: DB pointer 0x55b52dace000 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** DB Stats ** 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:39:12.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 2/0 79.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.2 0.00 0.00 1 0.000 0 0 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 79.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.2 0.00 0.00 1 0.000 0 0 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.2 0.00 0.00 1 0.000 0 0 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.2 0.00 0.00 1 0.000 0 0 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:39:12.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 25.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 25.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:39:12.117 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 2/0 79.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.2 0.00 0.00 1 0.000 0 0 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 79.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.2 0.00 0.00 1 0.000 0 0 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.2 0.00 0.00 1 0.000 0 0 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:39:12.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 25.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: starting mon.smithi032 rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi032 fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???) e1 preinit fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???).mds e1 new map 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???).mds e1 print_map 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: e1 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: legacy client fscid: -1 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:39:12.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: No filesystems configured 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 bash[24774]: 4da956fd8f670ef3a51012f8e9b434728a4c6a98172da7a7084ee96c2d500031 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 systemd[1]: Started Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: monmap e1: 1 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: fsmap 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:39:12.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:11 smithi032 ceph-mon[24793]: mgrmap e1: no daemons active 2024-03-24T11:39:13.070 INFO:teuthology.orchestra.run.smithi032.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:39:13.070 INFO:teuthology.orchestra.run.smithi032.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:39:13.070 INFO:teuthology.orchestra.run.smithi032.stderr:Creating mgr... 2024-03-24T11:39:13.070 INFO:teuthology.orchestra.run.smithi032.stderr:Verifying port 9283 ... 2024-03-24T11:39:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Failed to reset failed state of unit ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mgr.smithi032.xjdgbw.service: Unit ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mgr.smithi032.xjdgbw.service not loaded. 2024-03-24T11:39:13.247 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715.target.wants/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mgr.smithi032.xjdgbw.service → /etc/systemd/system/ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@.service. 2024-03-24T11:39:13.820 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:13 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/674809462' entity='client.admin' 2024-03-24T11:39:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: disabled 2024-03-24T11:39:13.943 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: inactive 2024-03-24T11:39:13.944 INFO:teuthology.orchestra.run.smithi032.stderr:firewalld.service is not enabled 2024-03-24T11:39:13.944 INFO:teuthology.orchestra.run.smithi032.stderr:Not possible to enable service . firewalld.service is not available 2024-03-24T11:39:13.953 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: disabled 2024-03-24T11:39:13.961 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: inactive 2024-03-24T11:39:13.962 INFO:teuthology.orchestra.run.smithi032.stderr:firewalld.service is not enabled 2024-03-24T11:39:13.962 INFO:teuthology.orchestra.run.smithi032.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-24T11:39:13.962 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for mgr to start... 2024-03-24T11:39:13.962 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for mgr... 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: { 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "fsid": "0ddfea52-e9d3-11ee-95cd-87774f69a715", 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "health": { 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T11:39:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 0 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "smithi032" 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T11:39:14.505 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T11:39:14.506 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T11:39:14.509 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "available": false, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T11:39:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "iostat", 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "nfs", 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "restful" 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "modified": "2024-03-24T11:39:07.523566+0000", 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T11:39:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: } 2024-03-24T11:39:14.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:14 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3331666017' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:39:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:mgr not available, waiting (1/15)... 2024-03-24T11:39:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: { 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "fsid": "0ddfea52-e9d3-11ee-95cd-87774f69a715", 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "health": { 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 0 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T11:39:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "smithi032" 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T11:39:17.561 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T11:39:17.562 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "available": false, 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "iostat", 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "nfs", 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "restful" 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:17.563 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "modified": "2024-03-24T11:39:07.523566+0000", 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T11:39:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: } 2024-03-24T11:39:18.128 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:17 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2137763929' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:39:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:mgr not available, waiting (2/15)... 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: Activating manager daemon smithi032.xjdgbw 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: mgrmap e2: smithi032.xjdgbw(active, starting, since 0.00281686s) 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:20.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:20.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:20.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:19 smithi032 ceph-mon[24793]: from='mgr.14100 172.21.15.32:0/1304154206' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:22.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:21 smithi032 ceph-mon[24793]: mgrmap e3: smithi032.xjdgbw(active, since 1.00707s) 2024-03-24T11:39:23.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:23 smithi032 ceph-mon[24793]: mgrmap e4: smithi032.xjdgbw(active, since 3s) 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: { 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "fsid": "0ddfea52-e9d3-11ee-95cd-87774f69a715", 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "health": { 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T11:39:24.425 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 0 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "smithi032" 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T11:39:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T11:39:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "available": true, 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "iostat", 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "nfs", 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "restful" 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ], 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:39:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T11:39:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:39:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "modified": "2024-03-24T11:39:07.523566+0000", 2024-03-24T11:39:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:39:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: }, 2024-03-24T11:39:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T11:39:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: } 2024-03-24T11:39:25.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:24 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3145816913' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:39:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:mgr is available 2024-03-24T11:39:29.291 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:29.291 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: [global] 2024-03-24T11:39:29.291 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: fsid = 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:29.291 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-24T11:39:29.291 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: [mgr] 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: [osd] 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-24T11:39:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-24T11:39:29.554 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:29 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/499786422' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:39:29.898 INFO:teuthology.orchestra.run.smithi032.stderr:Enabling cephadm module... 2024-03-24T11:39:30.927 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:30 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/777781470' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:39:31.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:31 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/777781470' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:39:31.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:31 smithi032 ceph-mon[24793]: mgrmap e5: smithi032.xjdgbw(active, since 10s) 2024-03-24T11:39:31.854 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: { 2024-03-24T11:39:31.855 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-24T11:39:31.855 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "available": true, 2024-03-24T11:39:31.855 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "active_name": "smithi032.xjdgbw", 2024-03-24T11:39:31.855 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-24T11:39:31.855 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: } 2024-03-24T11:39:32.462 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for the mgr to restart... 2024-03-24T11:39:32.463 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for mgr epoch 5... 2024-03-24T11:39:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:32 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2138133688' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:39:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: { 2024-03-24T11:39:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-24T11:39:37.691 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "initialized": true 2024-03-24T11:39:37.691 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: } 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: Active manager daemon smithi032.xjdgbw restarted 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: Activating manager daemon smithi032.xjdgbw 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: mgrmap e6: smithi032.xjdgbw(active, starting, since 0.0524298s) 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:39:37.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:37.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:37 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:38.223 INFO:teuthology.orchestra.run.smithi032.stderr:mgr epoch 5 is available 2024-03-24T11:39:38.223 INFO:teuthology.orchestra.run.smithi032.stderr:Setting orchestrator backend to cephadm... 2024-03-24T11:39:38.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:38 smithi032 ceph-mon[24793]: [24/Mar/2024:11:39:37] ENGINE Bus STARTING 2024-03-24T11:39:38.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:38 smithi032 ceph-mon[24793]: mgrmap e7: smithi032.xjdgbw(active, since 1.05464s) 2024-03-24T11:39:38.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:38 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:38.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:38 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:39.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:39 smithi032 ceph-mon[24793]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:39:39.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:39 smithi032 ceph-mon[24793]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:39:39.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:39 smithi032 ceph-mon[24793]: [24/Mar/2024:11:39:37] ENGINE Serving on https://172.21.15.32:7150 2024-03-24T11:39:39.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:39 smithi032 ceph-mon[24793]: [24/Mar/2024:11:39:37] ENGINE Bus STARTED 2024-03-24T11:39:39.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:39 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:39.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:39 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: value unchanged 2024-03-24T11:39:40.434 INFO:teuthology.orchestra.run.smithi032.stderr:Generating ssh key... 2024-03-24T11:39:40.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:40 smithi032 ceph-mon[24793]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:40.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:40 smithi032 ceph-mon[24793]: mgrmap e8: smithi032.xjdgbw(active, since 3s) 2024-03-24T11:39:41.946 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:41 smithi032 ceph-mon[24793]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:42.835 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:42 smithi032 ceph-mon[24793]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:42.835 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:42 smithi032 ceph-mon[24793]: Generating ssh key... 2024-03-24T11:39:42.835 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:42 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:42.835 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:42 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:42.835 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkkxU9AMJdCZJzMBWJd/DS66x8fbBYpIdSiiEdSxREX3UB1sN0Df+UeDxfo0YhilyJ1YgJ4qwkbpz9UrdpPG1beU6z3F/9o/nLMNTW1So58Namc7JSvDAEZYvHKMlTLT18oTyDZGTSkxchrQku41Amwi022QymCzr+qa45ziJHPqzUnpfJ5Zk1I8nnEhU8+tPiSooqFDZ8JIBo6MPDrIzSla4CtVip5P+qPfWwTMcI5FfS3Mary1/VTkRt1y09CEtfST8BZ8jBbjzx9Y0mgwNWWEaWpB8HcOM07LgC3J306y7B4svBaf8sA8PjQ5kidK+QOqGxxWpURuwtqlCVP81M6tFEhqgmNpSBh4UEeP1Oc7TDT8MsgVAO1kjaPvf0NqvF/ee0JWb59EnRlzEKvzVqDpGsSyNVjmIL357OP5KfiI0azG/gUurja9admsPDOxL7qZbiV5GSieLnVDQfwL3YTgr+LZ6fZpZe9tsFmMOfoK9jeM4sZSA3eYGX/JFqPqU= ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:39:43.339 INFO:teuthology.orchestra.run.smithi032.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:39:43.339 INFO:teuthology.orchestra.run.smithi032.stderr:Adding key to root@localhost authorized_keys... 2024-03-24T11:39:43.339 INFO:teuthology.orchestra.run.smithi032.stderr:Adding host smithi032... 2024-03-24T11:39:44.021 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:43 smithi032 ceph-mon[24793]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:44.995 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:44 smithi032 ceph-mon[24793]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:44.995 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:44 smithi032 ceph-mon[24793]: Deploying cephadm binary to smithi032 2024-03-24T11:39:45.002 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Added host 'smithi032' with addr '172.21.15.32' 2024-03-24T11:39:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:Deploying mon service with default placement... 2024-03-24T11:39:46.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:45 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:46.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:45 smithi032 ceph-mon[24793]: Added host smithi032 2024-03-24T11:39:46.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:45 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:46.209 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-24T11:39:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:Deploying mgr service with default placement... 2024-03-24T11:39:47.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:47 smithi032 ceph-mon[24793]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:47.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:47 smithi032 ceph-mon[24793]: Saving service mon spec with placement count:5 2024-03-24T11:39:47.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:47 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:47.604 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-24T11:39:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:Deploying crash service with default placement... 2024-03-24T11:39:48.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:48 smithi032 ceph-mon[24793]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:48.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:48 smithi032 ceph-mon[24793]: Saving service mgr spec with placement count:2 2024-03-24T11:39:48.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:48 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:49.100 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Scheduled crash update... 2024-03-24T11:39:49.821 INFO:teuthology.orchestra.run.smithi032.stderr:Deploying prometheus service with default placement... 2024-03-24T11:39:50.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:50 smithi032 ceph-mon[24793]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:50.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:50 smithi032 ceph-mon[24793]: Saving service crash spec with placement * 2024-03-24T11:39:50.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:50 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:50.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:50 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:50.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:50 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-03-24T11:39:51.613 INFO:teuthology.orchestra.run.smithi032.stderr:Deploying grafana service with default placement... 2024-03-24T11:39:51.778 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:51 smithi032 ceph-mon[24793]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:51.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:51 smithi032 ceph-mon[24793]: Saving service prometheus spec with placement count:1 2024-03-24T11:39:51.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:51 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:51.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:51 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-03-24T11:39:52.861 INFO:teuthology.orchestra.run.smithi032.stderr:Deploying node-exporter service with default placement... 2024-03-24T11:39:53.130 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:53 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:53.130 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:53 smithi032 ceph-mon[24793]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:53.130 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:53 smithi032 ceph-mon[24793]: Saving service grafana spec with placement count:1 2024-03-24T11:39:53.131 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:53 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-03-24T11:39:54.679 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:54 smithi032 ceph-mon[24793]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:54.679 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:54 smithi032 ceph-mon[24793]: Saving service node-exporter spec with placement * 2024-03-24T11:39:54.679 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:54 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:54.685 INFO:teuthology.orchestra.run.smithi032.stderr:Deploying alertmanager service with default placement... 2024-03-24T11:39:55.325 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-03-24T11:39:56.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:56 smithi032 ceph-mon[24793]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:56.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:56 smithi032 ceph-mon[24793]: Saving service alertmanager spec with placement count:1 2024-03-24T11:39:56.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:56 smithi032 ceph-mon[24793]: from='mgr.14118 172.21.15.32:0/4224128976' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:39:57.676 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:57 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2918507571' entity='client.admin' 2024-03-24T11:39:58.309 INFO:teuthology.orchestra.run.smithi032.stderr:Enabling the dashboard module... 2024-03-24T11:39:58.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:58 smithi032 ceph-mon[24793]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:39:58.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:58 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/630041235' entity='client.admin' 2024-03-24T11:39:59.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:39:59 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/626128466' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:40:00.911 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:00 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/626128466' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:40:00.911 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:00 smithi032 ceph-mon[24793]: mgrmap e9: smithi032.xjdgbw(active, since 23s) 2024-03-24T11:40:01.015 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: { 2024-03-24T11:40:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-24T11:40:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "available": true, 2024-03-24T11:40:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "active_name": "smithi032.xjdgbw", 2024-03-24T11:40:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-24T11:40:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: } 2024-03-24T11:40:01.623 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for the mgr to restart... 2024-03-24T11:40:01.623 INFO:teuthology.orchestra.run.smithi032.stderr:Waiting for mgr epoch 9... 2024-03-24T11:40:01.813 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:01 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3415010867' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:40:06.803 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: Active manager daemon smithi032.xjdgbw restarted 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: Activating manager daemon smithi032.xjdgbw 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: mgrmap e10: smithi032.xjdgbw(active, starting, since 0.0528516s) 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:06.804 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:40:06.805 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:40:06.805 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:06 smithi032 ceph-mon[24793]: [24/Mar/2024:11:40:06] ENGINE Bus STARTING 2024-03-24T11:40:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: { 2024-03-24T11:40:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-24T11:40:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: "initialized": true 2024-03-24T11:40:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: } 2024-03-24T11:40:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:mgr epoch 9 is available 2024-03-24T11:40:07.558 INFO:teuthology.orchestra.run.smithi032.stderr:Generating a dashboard self-signed certificate... 2024-03-24T11:40:08.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:07 smithi032 ceph-mon[24793]: [24/Mar/2024:11:40:06] ENGINE Serving on https://172.21.15.32:7150 2024-03-24T11:40:08.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:07 smithi032 ceph-mon[24793]: [24/Mar/2024:11:40:06] ENGINE Bus STARTED 2024-03-24T11:40:08.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:07 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:08.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:07 smithi032 ceph-mon[24793]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:40:08.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:07 smithi032 ceph-mon[24793]: mgrmap e11: smithi032.xjdgbw(active, since 1.05557s) 2024-03-24T11:40:08.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:07 smithi032 ceph-mon[24793]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:40:08.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:07 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:08.281 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-24T11:40:09.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:08 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:09.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:08 smithi032 ceph-mon[24793]: mgrmap e12: smithi032.xjdgbw(active, since 2s) 2024-03-24T11:40:09.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:08 smithi032 ceph-mon[24793]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:09.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:08 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:09.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:08 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:09.620 INFO:teuthology.orchestra.run.smithi032.stderr:Creating initial admin user... 2024-03-24T11:40:10.557 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$sNIzecdzxOf60dskGZqBmeSJhF3biXPMrH.Ai6b9sX8ethzmPjI8a", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711280410, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:40:11.200 INFO:teuthology.orchestra.run.smithi032.stderr:Fetching dashboard port number... 2024-03-24T11:40:11.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:11 smithi032 ceph-mon[24793]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:11.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:11 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: 8443 2024-03-24T11:40:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: disabled 2024-03-24T11:40:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:systemctl: inactive 2024-03-24T11:40:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:firewalld.service is not enabled 2024-03-24T11:40:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:40:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:Ceph Dashboard is now available at: 2024-03-24T11:40:12.459 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:12.459 INFO:teuthology.orchestra.run.smithi032.stderr: URL: https://smithi032.front.sepia.ceph.com:8443/ 2024-03-24T11:40:12.459 INFO:teuthology.orchestra.run.smithi032.stderr: User: admin 2024-03-24T11:40:12.459 INFO:teuthology.orchestra.run.smithi032.stderr: Password: yf5anvjv2d 2024-03-24T11:40:12.459 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:Enabling autotune for osd_memory_target 2024-03-24T11:40:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:12 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1291618267' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:40:13.906 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:13 smithi032 ceph-mon[24793]: mgrmap e13: smithi032.xjdgbw(active, since 6s) 2024-03-24T11:40:14.391 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-24T11:40:14.938 INFO:teuthology.orchestra.run.smithi032.stderr:You can access the Ceph CLI with: 2024-03-24T11:40:14.938 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:14.938 INFO:teuthology.orchestra.run.smithi032.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:14.938 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:14.938 INFO:teuthology.orchestra.run.smithi032.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:40:14.938 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:14.938 INFO:teuthology.orchestra.run.smithi032.stderr: ceph telemetry on 2024-03-24T11:40:14.939 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:14.939 INFO:teuthology.orchestra.run.smithi032.stderr:For more information see: 2024-03-24T11:40:14.939 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:14.939 INFO:teuthology.orchestra.run.smithi032.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-24T11:40:14.939 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T11:40:14.939 INFO:teuthology.orchestra.run.smithi032.stderr:Bootstrap complete. 2024-03-24T11:40:14.978 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:40:14.981 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:40:14.981 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:40:15.001 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:40:15.001 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:40:15.001 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:40:15.058 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:40:15.058 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:40:15.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/keyring of=/dev/stdout 2024-03-24T11:40:15.126 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:40:15.126 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:40:15.126 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:40:15.181 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:40:15.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkkxU9AMJdCZJzMBWJd/DS66x8fbBYpIdSiiEdSxREX3UB1sN0Df+UeDxfo0YhilyJ1YgJ4qwkbpz9UrdpPG1beU6z3F/9o/nLMNTW1So58Namc7JSvDAEZYvHKMlTLT18oTyDZGTSkxchrQku41Amwi022QymCzr+qa45ziJHPqzUnpfJ5Zk1I8nnEhU8+tPiSooqFDZ8JIBo6MPDrIzSla4CtVip5P+qPfWwTMcI5FfS3Mary1/VTkRt1y09CEtfST8BZ8jBbjzx9Y0mgwNWWEaWpB8HcOM07LgC3J306y7B4svBaf8sA8PjQ5kidK+QOqGxxWpURuwtqlCVP81M6tFEhqgmNpSBh4UEeP1Oc7TDT8MsgVAO1kjaPvf0NqvF/ee0JWb59EnRlzEKvzVqDpGsSyNVjmIL357OP5KfiI0azG/gUurja9admsPDOxL7qZbiV5GSieLnVDQfwL3YTgr+LZ6fZpZe9tsFmMOfoK9jeM4sZSA3eYGX/JFqPqU= ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:40:15.260 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkkxU9AMJdCZJzMBWJd/DS66x8fbBYpIdSiiEdSxREX3UB1sN0Df+UeDxfo0YhilyJ1YgJ4qwkbpz9UrdpPG1beU6z3F/9o/nLMNTW1So58Namc7JSvDAEZYvHKMlTLT18oTyDZGTSkxchrQku41Amwi022QymCzr+qa45ziJHPqzUnpfJ5Zk1I8nnEhU8+tPiSooqFDZ8JIBo6MPDrIzSla4CtVip5P+qPfWwTMcI5FfS3Mary1/VTkRt1y09CEtfST8BZ8jBbjzx9Y0mgwNWWEaWpB8HcOM07LgC3J306y7B4svBaf8sA8PjQ5kidK+QOqGxxWpURuwtqlCVP81M6tFEhqgmNpSBh4UEeP1Oc7TDT8MsgVAO1kjaPvf0NqvF/ee0JWb59EnRlzEKvzVqDpGsSyNVjmIL357OP5KfiI0azG/gUurja9admsPDOxL7qZbiV5GSieLnVDQfwL3YTgr+LZ6fZpZe9tsFmMOfoK9jeM4sZSA3eYGX/JFqPqU= ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:40:15.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkkxU9AMJdCZJzMBWJd/DS66x8fbBYpIdSiiEdSxREX3UB1sN0Df+UeDxfo0YhilyJ1YgJ4qwkbpz9UrdpPG1beU6z3F/9o/nLMNTW1So58Namc7JSvDAEZYvHKMlTLT18oTyDZGTSkxchrQku41Amwi022QymCzr+qa45ziJHPqzUnpfJ5Zk1I8nnEhU8+tPiSooqFDZ8JIBo6MPDrIzSla4CtVip5P+qPfWwTMcI5FfS3Mary1/VTkRt1y09CEtfST8BZ8jBbjzx9Y0mgwNWWEaWpB8HcOM07LgC3J306y7B4svBaf8sA8PjQ5kidK+QOqGxxWpURuwtqlCVP81M6tFEhqgmNpSBh4UEeP1Oc7TDT8MsgVAO1kjaPvf0NqvF/ee0JWb59EnRlzEKvzVqDpGsSyNVjmIL357OP5KfiI0azG/gUurja9admsPDOxL7qZbiV5GSieLnVDQfwL3YTgr+LZ6fZpZe9tsFmMOfoK9jeM4sZSA3eYGX/JFqPqU= ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:40:15.321 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkkxU9AMJdCZJzMBWJd/DS66x8fbBYpIdSiiEdSxREX3UB1sN0Df+UeDxfo0YhilyJ1YgJ4qwkbpz9UrdpPG1beU6z3F/9o/nLMNTW1So58Namc7JSvDAEZYvHKMlTLT18oTyDZGTSkxchrQku41Amwi022QymCzr+qa45ziJHPqzUnpfJ5Zk1I8nnEhU8+tPiSooqFDZ8JIBo6MPDrIzSla4CtVip5P+qPfWwTMcI5FfS3Mary1/VTkRt1y09CEtfST8BZ8jBbjzx9Y0mgwNWWEaWpB8HcOM07LgC3J306y7B4svBaf8sA8PjQ5kidK+QOqGxxWpURuwtqlCVP81M6tFEhqgmNpSBh4UEeP1Oc7TDT8MsgVAO1kjaPvf0NqvF/ee0JWb59EnRlzEKvzVqDpGsSyNVjmIL357OP5KfiI0azG/gUurja9admsPDOxL7qZbiV5GSieLnVDQfwL3YTgr+LZ6fZpZe9tsFmMOfoK9jeM4sZSA3eYGX/JFqPqU= ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:40:15.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:40:15.682 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:15 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/717758927' entity='client.admin' 2024-03-24T11:40:18.021 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:17 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:18.022 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:17 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:18.022 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:17 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:18.022 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:17 smithi032 ceph-mon[24793]: Deploying daemon alertmanager.smithi032 on smithi032 2024-03-24T11:40:19.268 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:40:19.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:40:19.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:19 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/452237832' entity='client.admin' 2024-03-24T11:40:22.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:22 smithi032 ceph-mon[24793]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:22.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:22 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:23.122 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2024-03-24T11:40:23.122 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-24T11:40:23.122 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:40:23.140 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-24T11:40:23.141 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:23.197 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2024-03-24T11:40:23.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch host add smithi146 2024-03-24T11:40:26.921 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:26 smithi032 ceph-mon[24793]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:28.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:27 smithi032 ceph-mon[24793]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:28.463 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi146' with addr '172.21.15.146' 2024-03-24T11:40:29.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:29.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: Deploying cephadm binary to smithi146 2024-03-24T11:40:29.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:29.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:40:29.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T11:40:29.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:29.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: Deploying daemon crash.smithi032 on smithi032 2024-03-24T11:40:29.286 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:29.286 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:28 smithi032 ceph-mon[24793]: Added host smithi146 2024-03-24T11:40:29.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:40:31.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:30 smithi032 ceph-mon[24793]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:31.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:30 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:31.644 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:40:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}] 2024-03-24T11:40:32.083 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:31 smithi032 ceph-mon[24793]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:40:32.083 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:31 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:32.430 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:40:32.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:40:33.287 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:33 smithi032 ceph-mon[24793]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:33.287 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:33 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:33.287 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:33 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:33.287 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:33 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:33.287 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:33 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:33.288 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:33 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:33.288 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:33 smithi032 ceph-mon[24793]: Deploying daemon grafana.smithi032 on smithi032 2024-03-24T11:40:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-03-24T11:40:35.353 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:35 smithi032 ceph-mon[24793]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:35.353 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:35 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2728492974' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:40:35.677 INFO:tasks.cephadm:Adding mon.smithi032 on smithi032 2024-03-24T11:40:35.677 INFO:tasks.cephadm:Adding mon.smithi146 on smithi146 2024-03-24T11:40:35.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply mon '2;smithi032:172.21.15.32=smithi032;smithi146:172.21.15.146=smithi146' 2024-03-24T11:40:36.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:36 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2728492974' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:40:36.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:36 smithi032 ceph-mon[24793]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:40:36.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:36 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:37.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:37 smithi032 ceph-mon[24793]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:37.525 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mon update... 2024-03-24T11:40:38.105 DEBUG:teuthology.orchestra.run.smithi146:mon.smithi146> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi146.service 2024-03-24T11:40:38.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:40:38.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:40:38.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:38 smithi032 ceph-mon[24793]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi032:172.21.15.32=smithi032;smithi146:172.21.15.146=smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:38.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:38 smithi032 ceph-mon[24793]: Saving service mon spec with placement smithi032:172.21.15.32=smithi032;smithi146:172.21.15.146=smithi146;count:2 2024-03-24T11:40:38.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:38 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:39 smithi032 ceph-mon[24793]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:40:40.048 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:40:40.050 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:40:40.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:40 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/80139874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:40:41.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:40:41.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:40:41.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:41 smithi032 ceph-mon[24793]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:43.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:40:43.442 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:40:43.444 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:40:43.798 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:43 smithi032 ceph-mon[24793]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:43.798 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:43 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/2009797774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:40:44.993 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:40:44.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:40:45.661 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:45 smithi032 ceph-mon[24793]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:46.828 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:40:46.828 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:40:46.830 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:40:47.553 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:47 smithi032 ceph-mon[24793]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:47.554 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:47 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:47.554 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:47 smithi032 ceph-mon[24793]: Deploying daemon node-exporter.smithi032 on smithi032 2024-03-24T11:40:47.554 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:47 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1535629847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:40:48.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:40:48.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:40:49.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:49 smithi032 ceph-mon[24793]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:40:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:40:50.299 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:40:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:40:50.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:50 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:50.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:50 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:40:50.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:50 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/3187375043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:40:51.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:40:51.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:40:51.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:51 smithi032 ceph-mon[24793]: from='mgr.14160 172.21.15.32:0/1417033370' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:40:51.938 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:51 smithi032 ceph-mon[24793]: mgrmap e14: smithi032.xjdgbw(active, since 44s) 2024-03-24T11:40:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:40:53.483 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:40:53.485 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:40:53.874 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:53 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/730623265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:40:55.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:40:55.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:40:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:40:56.864 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:40:56.865 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:40:57.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: Active manager daemon smithi032.xjdgbw restarted 2024-03-24T11:40:57.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: Activating manager daemon smithi032.xjdgbw 2024-03-24T11:40:57.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T11:40:57.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: mgrmap e15: smithi032.xjdgbw(active, starting, since 0.052336s) 2024-03-24T11:40:57.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:40:57.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:40:57.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:40:57.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:40:57.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:40:57.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:40:57.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:57.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:57.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:57.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:40:57.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:57.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:40:57.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:40:57.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/342402312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:40:57.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:58.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:40:58.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:40:58.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:58 smithi032 ceph-mon[24793]: mgrmap e16: smithi032.xjdgbw(active, since 1.05476s) 2024-03-24T11:40:58.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:58 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:58.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:58 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:59.683 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:59 smithi032 ceph-mon[24793]: [24/Mar/2024:11:40:58] ENGINE Bus STARTING 2024-03-24T11:40:59.683 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:59 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:40:59.683 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:40:59 smithi032 ceph-mon[24793]: mgrmap e17: smithi032.xjdgbw(active, since 2s) 2024-03-24T11:41:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:41:00.085 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:41:00.087 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:41:00.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:00 smithi032 ceph-mon[24793]: [24/Mar/2024:11:40:58] ENGINE Serving on https://172.21.15.32:7150 2024-03-24T11:41:00.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:00 smithi032 ceph-mon[24793]: [24/Mar/2024:11:40:58] ENGINE Bus STARTED 2024-03-24T11:41:00.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:00 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1388412610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:00.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:00 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:02.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:41:02.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:41:02.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:02.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:02.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:01 smithi032 ceph-mon[24793]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:41:02.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:01 smithi032 ceph-mon[24793]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:02.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:03 smithi032 ceph-mon[24793]: Deploying daemon crash.smithi146 on smithi146 2024-03-24T11:41:04.428 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:41:04.428 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:41:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:41:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:04 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/877646574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:06.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:41:06.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:41:07.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:07.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:07.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:41:07.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:41:07.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:07.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:07 smithi032 ceph-mon[24793]: Deploying daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:41:07.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:41:09.603 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:41:09.605 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:41:09.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:09 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/330132201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:10.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:10.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:41:10.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:10.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:10 smithi032 ceph-mon[24793]: Deploying daemon mon.smithi146 on smithi146 2024-03-24T11:41:11.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:41:11.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:41:13.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:12 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:41:14.449 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:39:04.755364Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:41:14.450 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-24T11:41:14.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:14 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/465291688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:16.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:41:16.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: mon.smithi032 calling monitor election 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/1649041082' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: mon.smithi146 calling monitor election 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: mon.smithi032 is new leader, mons smithi032,smithi146 in quorum (ranks 0,1) 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi146=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: fsmap 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: mgrmap e17: smithi032.xjdgbw(active, since 23s) 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/1649041082' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: overall HEALTH_OK 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/1649041082' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:41:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/1649041082' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:20.705 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-24T11:41:21.454 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:41:15.151694Z","created":"2024-03-24T11:39:04.755364Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi146","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:41:21.454 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 2 2024-03-24T11:41:21.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:21 smithi032 ceph-mon[24793]: Deploying daemon node-exporter.smithi146 on smithi146 2024-03-24T11:41:21.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:21 smithi032 ceph-mon[24793]: mgrmap e18: smithi032.xjdgbw(active, since 23s), standbys: smithi146.tfohfl 2024-03-24T11:41:21.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:21 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi146.tfohfl", "id": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:41:21.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:21 smithi032 ceph-mon[24793]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:21.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:21 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:41:22.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:22 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1969948442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:22.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:22 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:23.083 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:41:23.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:41:23.391 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:23 smithi032 ceph-mon[24793]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:24.756 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:41:24.757 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-24T11:41:24.757 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:41:24.757 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2024-03-24T11:41:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:24 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/4039353257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:25.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:25.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:24 smithi032 ceph-mon[24793]: Deploying daemon prometheus.smithi032 on smithi032 2024-03-24T11:41:25.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:24 smithi032 ceph-mon[24793]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:25.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:24 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/4039353257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:25.651 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:41:25.651 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:41:25.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:41:25.683 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:41:25.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:25.764 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-24T11:41:25.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:41:25.800 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-24T11:41:25.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:25.870 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:41:25.871 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:41:25.871 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:41:25.887 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:41:25.887 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-24 11:40:46.855213550 +0000 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-24 11:38:14.684828411 +0000 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-24 11:38:14.684828411 +0000 2024-03-24T11:41:25.946 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-24 11:38:14.684828411 +0000 2024-03-24T11:41:25.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:41:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-24T11:41:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-24T11:41:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000102399 s, 5.0 MB/s 2024-03-24T11:41:26.017 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:41:26.074 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-24 11:40:46.855213550 +0000 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-24 11:38:15.015822110 +0000 2024-03-24T11:41:26.130 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-24 11:38:15.015822110 +0000 2024-03-24T11:41:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-24 11:38:15.015822110 +0000 2024-03-24T11:41:26.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:41:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-24T11:41:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-24T11:41:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.8732e-05 s, 5.2 MB/s 2024-03-24T11:41:26.202 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:41:26.258 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-03-24T11:41:26.314 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-24 11:40:46.855213550 +0000 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-24 11:38:15.334816038 +0000 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-24 11:38:15.334816038 +0000 2024-03-24T11:41:26.315 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-24 11:38:15.334816038 +0000 2024-03-24T11:41:26.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:41:26.382 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-24T11:41:26.382 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-24T11:41:26.383 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.145e-05 s, 5.6 MB/s 2024-03-24T11:41:26.384 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:41:26.439 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-03-24T11:41:26.493 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:41:26.493 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:26.493 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T11:41:26.493 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:26.493 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:26.493 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-24 11:40:46.856213531 +0000 2024-03-24T11:41:26.493 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-24 11:38:15.663809775 +0000 2024-03-24T11:41:26.494 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-24 11:38:15.663809775 +0000 2024-03-24T11:41:26.494 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-24 11:38:15.663809775 +0000 2024-03-24T11:41:26.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:41:26.558 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-24T11:41:26.558 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-24T11:41:26.558 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.9898e-05 s, 5.1 MB/s 2024-03-24T11:41:26.559 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:41:26.615 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-24T11:41:26.615 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:41:26.633 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:41:26.634 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-03-24T11:41:26.691 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-24 11:41:20.789294960 +0000 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-24 11:38:14.235381314 +0000 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-24 11:38:14.235381314 +0000 2024-03-24T11:41:26.692 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-24 11:38:14.235381314 +0000 2024-03-24T11:41:26.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:41:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-24T11:41:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-24T11:41:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000208935 s, 2.5 MB/s 2024-03-24T11:41:26.763 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:41:26.821 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-24 11:41:20.789294960 +0000 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-24 11:38:14.582375495 +0000 2024-03-24T11:41:26.878 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-24 11:38:14.582375495 +0000 2024-03-24T11:41:26.879 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-24 11:38:14.582375495 +0000 2024-03-24T11:41:26.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:41:26.946 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-24T11:41:26.946 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-24T11:41:26.947 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000228517 s, 2.2 MB/s 2024-03-24T11:41:26.948 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:41:27.006 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-24 11:41:20.790294944 +0000 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-24 11:38:14.925369744 +0000 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-24 11:38:14.925369744 +0000 2024-03-24T11:41:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-24 11:38:14.925369744 +0000 2024-03-24T11:41:27.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:41:27.128 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:26 smithi146 ceph-mon[26954]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:27.129 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:26 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:27.131 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-24T11:41:27.131 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-24T11:41:27.131 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000219343 s, 2.3 MB/s 2024-03-24T11:41:27.132 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:41:27.189 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-03-24T11:41:27.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:26 smithi032 ceph-mon[24793]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:27.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:26 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:27.245 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-24 11:41:20.790294944 +0000 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-24 11:38:15.276363859 +0000 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-24 11:38:15.276363859 +0000 2024-03-24T11:41:27.246 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-24 11:38:15.276363859 +0000 2024-03-24T11:41:27.247 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:41:27.313 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-24T11:41:27.313 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-24T11:41:27.313 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000221218 s, 2.3 MB/s 2024-03-24T11:41:27.314 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:41:27.371 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/vg_nvme/lv_4... 2024-03-24T11:41:27.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:41:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:28 smithi146 ceph-mon[26954]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:29.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:28 smithi032 ceph-mon[24793]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:30.077 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:41:30.077 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:41:30.077 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-03-24T11:41:30.077 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-03-24T11:41:30.077 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00887941 s, 1.2 GB/s 2024-03-24T11:41:30.077 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-03-24T11:41:31.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:30 smithi146 ceph-mon[26954]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:31.179 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:30 smithi032 ceph-mon[24793]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:31.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-03-24T11:41:33.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:32 smithi032 ceph-mon[24793]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:33.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:32 smithi146 ceph-mon[26954]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:34.818 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:34 smithi032 ceph-mon[24793]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:34 smithi146 ceph-mon[26954]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:36.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:36 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:41:36.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:36 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:41:36.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:36 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:36.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:36 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:41:36.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:36 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:41:36.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:36 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:37.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:37 smithi032 ceph-mon[24793]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:37.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:37 smithi032 ceph-mon[24793]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:37.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:37.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:37.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:37.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:37.817 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:37 smithi146 ceph-mon[26954]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:37.817 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:37 smithi146 ceph-mon[26954]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:37.817 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:37.817 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:37.817 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:37.817 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:39.006 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:38 smithi146 ceph-mon[26954]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:39.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:38 smithi032 ceph-mon[24793]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:40.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:40.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:41:40.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/509141058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ac184de-5c35-4015-b99e-75966907f527"}]: dispatch 2024-03-24T11:41:40.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/509141058' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ac184de-5c35-4015-b99e-75966907f527"}]': finished 2024-03-24T11:41:40.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:41:40.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:40.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:40.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:40.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:39 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/4285912654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:41:40.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/509141058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ac184de-5c35-4015-b99e-75966907f527"}]: dispatch 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/509141058' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ac184de-5c35-4015-b99e-75966907f527"}]': finished 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:40.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:39 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/4285912654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:41:41.037 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:40 smithi146 ceph-mon[26954]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:41.059 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:40 smithi032 ceph-mon[24793]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: Reconfiguring mon.smithi032 (unknown last config time)... 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: Reconfiguring daemon mon.smithi032 on smithi032 2024-03-24T11:41:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:42 smithi146 ceph-mon[26954]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:43.158 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.158 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.158 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: Reconfiguring mon.smithi032 (unknown last config time)... 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: Reconfiguring daemon mon.smithi032 on smithi032 2024-03-24T11:41:43.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:42 smithi032 ceph-mon[24793]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:44.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:44.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:44 smithi032 ceph-mon[24793]: Reconfiguring mgr.smithi032.xjdgbw (unknown last config time)... 2024-03-24T11:41:44.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:44.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:41:44.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:44.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:44 smithi032 ceph-mon[24793]: Reconfiguring daemon mgr.smithi032.xjdgbw on smithi032 2024-03-24T11:41:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:44 smithi146 ceph-mon[26954]: Reconfiguring mgr.smithi032.xjdgbw (unknown last config time)... 2024-03-24T11:41:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:41:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:44 smithi146 ceph-mon[26954]: Reconfiguring daemon mgr.smithi032.xjdgbw on smithi032 2024-03-24T11:41:45.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:45 smithi032 ceph-mon[24793]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:45 smithi146 ceph-mon[26954]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:46.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:46 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:46.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:46 smithi032 ceph-mon[24793]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-03-24T11:41:46.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:46 smithi032 ceph-mon[24793]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-03-24T11:41:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:46 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:46 smithi146 ceph-mon[26954]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-03-24T11:41:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:46 smithi146 ceph-mon[26954]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-03-24T11:41:47.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:47 smithi032 ceph-mon[24793]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:47.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:47 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:41:47.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:47 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:47 smithi146 ceph-mon[26954]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:47 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:41:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:47 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:48.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:48 smithi032 ceph-mon[24793]: Deploying daemon osd.0 on smithi032 2024-03-24T11:41:49.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:48 smithi146 ceph-mon[26954]: Deploying daemon osd.0 on smithi032 2024-03-24T11:41:49.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:49 smithi032 ceph-mon[24793]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:50.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:49 smithi146 ceph-mon[26954]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:50.788 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:50 smithi032 ceph-mon[24793]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:50 smithi146 ceph-mon[26954]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:52.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:51 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:52.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:51 smithi032 ceph-mon[24793]: Reconfiguring crash.smithi032 (monmap changed)... 2024-03-24T11:41:52.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:51 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:41:52.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:51 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:52.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:51 smithi032 ceph-mon[24793]: Reconfiguring daemon crash.smithi032 on smithi032 2024-03-24T11:41:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:51 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:51 smithi146 ceph-mon[26954]: Reconfiguring crash.smithi032 (monmap changed)... 2024-03-24T11:41:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:51 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:41:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:51 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:51 smithi146 ceph-mon[26954]: Reconfiguring daemon crash.smithi032 on smithi032 2024-03-24T11:41:53.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:52 smithi032 ceph-mon[24793]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:53.267 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:52 smithi146 ceph-mon[26954]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:55.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:54 smithi146 ceph-mon[26954]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:55.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:54 smithi032 ceph-mon[24793]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:57.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:57.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:56 smithi032 ceph-mon[24793]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:56 smithi146 ceph-mon[26954]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:57.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:57.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:57.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:57.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:57 smithi032 ceph-mon[24793]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-03-24T11:41:57.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:57 smithi032 ceph-mon[24793]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-03-24T11:41:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:41:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:57 smithi146 ceph-mon[26954]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-03-24T11:41:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:57 smithi146 ceph-mon[26954]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-03-24T11:41:59.061 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:58 smithi032 ceph-mon[24793]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:41:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:58 smithi146 ceph-mon[26954]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:42:00.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:41:59 smithi146 ceph-mon[26954]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:42:00.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:41:59 smithi032 ceph-mon[24793]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:42:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:00 smithi146 ceph-mon[26954]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:42:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:00 smithi146 ceph-mon[26954]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:42:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:00 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:00 smithi146 ceph-mon[26954]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:42:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:00 smithi146 ceph-mon[26954]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:42:01.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:00 smithi032 ceph-mon[24793]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:42:01.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:00 smithi032 ceph-mon[24793]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:42:01.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:00 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:01.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:00 smithi032 ceph-mon[24793]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:42:01.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:00 smithi032 ceph-mon[24793]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:42:02.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:01 smithi032 ceph-mon[24793]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:42:02.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:01 smithi032 ceph-mon[24793]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T11:42:02.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:02.055 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:02.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:01 smithi146 ceph-mon[26954]: from='osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:42:02.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:01 smithi146 ceph-mon[26954]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T11:42:02.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:01 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:02.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:01 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652] boot 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: Reconfiguring crash.smithi146 (monmap changed)... 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:02.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: Reconfiguring daemon crash.smithi146 on smithi146 2024-03-24T11:42:02.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:02 smithi032 ceph-mon[24793]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:42:03.029 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:42:03.029 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:42:03.029 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: osd.0 [v2:172.21.15.32:6802/249424652,v1:172.21.15.32:6803/249424652] boot 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: Reconfiguring crash.smithi146 (monmap changed)... 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: Reconfiguring daemon crash.smithi146 on smithi146 2024-03-24T11:42:03.030 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:02 smithi146 ceph-mon[26954]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:42:03.967 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:03 smithi146 ceph-mon[26954]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T11:42:04.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:03 smithi032 ceph-mon[24793]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T11:42:04.086 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 0 on host 'smithi032' 2024-03-24T11:42:04.653 DEBUG:teuthology.orchestra.run.smithi032:osd.0> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:42:04.656 INFO:tasks.cephadm:Deploying osd.1 on smithi032 with /dev/vg_nvme/lv_3... 2024-03-24T11:42:04.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:42:05.313 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:05.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: Reconfiguring mgr.smithi146.tfohfl (monmap changed)... 2024-03-24T11:42:05.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:05.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:05.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:05.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: Reconfiguring daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:42:05.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:05.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:05 smithi032 ceph-mon[24793]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:05.373 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:05.374 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: Reconfiguring mgr.smithi146.tfohfl (monmap changed)... 2024-03-24T11:42:05.374 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:05.374 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:05.374 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:05.374 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: Reconfiguring daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:42:05.375 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:05.375 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:05 smithi146 ceph-mon[26954]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:06.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:06.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:06 smithi032 ceph-mon[24793]: Reconfiguring mon.smithi146 (monmap changed)... 2024-03-24T11:42:06.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:06.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:06.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:06.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:06 smithi032 ceph-mon[24793]: Reconfiguring daemon mon.smithi146 on smithi146 2024-03-24T11:42:06.661 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:42:06.661 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:42:06.661 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-03-24T11:42:06.662 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-03-24T11:42:06.662 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00837206 s, 1.3 GB/s 2024-03-24T11:42:06.662 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-03-24T11:42:06.716 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:06.716 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:06 smithi146 ceph-mon[26954]: Reconfiguring mon.smithi146 (monmap changed)... 2024-03-24T11:42:06.717 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:06.717 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:06.717 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:06.717 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:06 smithi146 ceph-mon[26954]: Reconfiguring daemon mon.smithi146 on smithi146 2024-03-24T11:42:07.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.32:9093"}]: dispatch 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.32:3000"}]: dispatch 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.32:9095"}]: dispatch 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.32:9093"}]: dispatch 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.32:3000"}]: dispatch 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:42:07.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.32:9095"}]: dispatch 2024-03-24T11:42:07.654 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:07.654 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:07.654 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:07.654 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:08.490 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:08 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:42:08.490 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:08 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.32:9093"}]: dispatch 2024-03-24T11:42:08.490 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:08 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:42:08.490 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:08 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.32:3000"}]: dispatch 2024-03-24T11:42:08.490 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:08 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:42:08.490 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:08 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.32:9095"}]: dispatch 2024-03-24T11:42:08.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:08 smithi146 ceph-mon[26954]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:42:08.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:08 smithi146 ceph-mon[26954]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.32:9093"}]: dispatch 2024-03-24T11:42:08.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:08 smithi146 ceph-mon[26954]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:42:08.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:08 smithi146 ceph-mon[26954]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.32:3000"}]: dispatch 2024-03-24T11:42:08.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:08 smithi146 ceph-mon[26954]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:42:08.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:08 smithi146 ceph-mon[26954]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.32:9095"}]: dispatch 2024-03-24T11:42:09.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:09 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:09.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:09 smithi032 ceph-mon[24793]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:09.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:09 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:09.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:09 smithi146 ceph-mon[26954]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:10.671 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:42:10.671 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:42:10.671 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:10.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:10 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:42:10.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:10 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:42:10.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:10 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:11.786 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:11 smithi146 ceph-mon[26954]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:11.786 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:11 smithi146 ceph-mon[26954]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:11.812 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:11 smithi032 ceph-mon[24793]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:11.812 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:11 smithi032 ceph-mon[24793]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:13.300 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:12 smithi032 ceph-mon[24793]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:13.300 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:12 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1318751249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4ec7af-f4dd-4201-aae9-cb21009558dc"}]: dispatch 2024-03-24T11:42:13.300 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:12 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1318751249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d4ec7af-f4dd-4201-aae9-cb21009558dc"}]': finished 2024-03-24T11:42:13.300 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:12 smithi032 ceph-mon[24793]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:42:13.300 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:12 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:12 smithi146 ceph-mon[26954]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:12 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1318751249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4ec7af-f4dd-4201-aae9-cb21009558dc"}]: dispatch 2024-03-24T11:42:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:12 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1318751249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d4ec7af-f4dd-4201-aae9-cb21009558dc"}]': finished 2024-03-24T11:42:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:12 smithi146 ceph-mon[26954]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:42:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:12 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:14.083 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:13 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3709751663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:42:14.083 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:13 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:14.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:13 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3709751663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:42:14.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:13 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:15.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:14 smithi032 ceph-mon[24793]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:15.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:14 smithi146 ceph-mon[26954]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:17.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:16 smithi032 ceph-mon[24793]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:17.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:16 smithi146 ceph-mon[26954]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: Detected new or changed devices on smithi032 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi032 to 12869M 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:19.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:19 smithi032 ceph-mon[24793]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: Detected new or changed devices on smithi032 2024-03-24T11:42:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi032 to 12869M 2024-03-24T11:42:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:19.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:19.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:19.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:19 smithi146 ceph-mon[26954]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:20.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:20 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:42:20.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:20 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:20.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:20 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:20.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:42:20.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:20.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:20 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:21.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:21 smithi032 ceph-mon[24793]: Deploying daemon osd.1 on smithi032 2024-03-24T11:42:21.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:21 smithi032 ceph-mon[24793]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:21.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:21 smithi146 ceph-mon[26954]: Deploying daemon osd.1 on smithi032 2024-03-24T11:42:21.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:21 smithi146 ceph-mon[26954]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:23.153 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:22 smithi032 ceph-mon[24793]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:23.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:22 smithi146 ceph-mon[26954]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:24 smithi146 ceph-mon[26954]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:25.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:24 smithi032 ceph-mon[24793]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:26.749 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:26 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:26.750 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:26 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:27.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:26 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:27.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:26 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:27.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:27 smithi146 ceph-mon[26954]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:27.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:27 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:27.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:27 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:27.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:27 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:27.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:27 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:27.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:27 smithi032 ceph-mon[24793]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:27.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:27 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:27.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:27 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:27.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:27 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:27.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:27 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:29.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:28 smithi032 ceph-mon[24793]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:29.074 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:28 smithi146 ceph-mon[26954]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:30.362 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:30 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:30.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:30 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:30.395 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 1 on host 'smithi032' 2024-03-24T11:42:30.989 DEBUG:teuthology.orchestra.run.smithi032:osd.1> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:42:30.992 INFO:tasks.cephadm:Deploying osd.2 on smithi032 with /dev/vg_nvme/lv_2... 2024-03-24T11:42:30.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:42:31.697 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:31.698 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:31 smithi032 ceph-mon[24793]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:42:31.698 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:31 smithi032 ceph-mon[24793]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:31.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:31.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:31 smithi146 ceph-mon[26954]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:42:31.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:31 smithi146 ceph-mon[26954]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:32.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:32 smithi032 ceph-mon[24793]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:42:32.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:32 smithi032 ceph-mon[24793]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:42:32.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:32 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:32.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:32 smithi032 ceph-mon[24793]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:42:32.611 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:42:32 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[39236]: 2024-03-24T11:42:32.401+0000 7f3f70bbc700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T11:42:32.611 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:42:32 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[39236]: 2024-03-24T11:42:32.404+0000 7f3f6cd56700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:42:32.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:32 smithi146 ceph-mon[26954]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:42:32.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:32 smithi146 ceph-mon[26954]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:42:32.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:32 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:32.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:32 smithi146 ceph-mon[26954]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:42:33.559 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:33 smithi032 ceph-mon[24793]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:42:33.559 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:33 smithi032 ceph-mon[24793]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T11:42:33.559 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:33 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:33.559 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:33 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:33.559 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:33 smithi032 ceph-mon[24793]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:33.559 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:33 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:42:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:42:33.710 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-03-24T11:42:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-03-24T11:42:33.710 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00971575 s, 1.1 GB/s 2024-03-24T11:42:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-03-24T11:42:33.713 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:33 smithi146 ceph-mon[26954]: from='osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:42:33.713 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:33 smithi146 ceph-mon[26954]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T11:42:33.713 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:33 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:33.713 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:33 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:33.714 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:33 smithi146 ceph-mon[26954]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:42:33.714 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:33 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:34.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-03-24T11:42:34.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:34 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:42:34.603 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:34 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:42:34.603 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:34 smithi032 ceph-mon[24793]: osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682] boot 2024-03-24T11:42:34.603 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:34 smithi032 ceph-mon[24793]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:42:34.603 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:34 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:34 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:42:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:34 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:42:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:34 smithi146 ceph-mon[26954]: osd.1 [v2:172.21.15.32:6810/3261903682,v1:172.21.15.32:6811/3261903682] boot 2024-03-24T11:42:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:34 smithi146 ceph-mon[26954]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:42:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:34 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:42:35.554 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:35 smithi032 ceph-mon[24793]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T11:42:35.554 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:35 smithi032 ceph-mon[24793]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:35 smithi146 ceph-mon[26954]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T11:42:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:35 smithi146 ceph-mon[26954]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:37.016 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:36 smithi032 ceph-mon[24793]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:36 smithi146 ceph-mon[26954]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:38.347 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:38 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:38.347 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:38 smithi032 ceph-mon[24793]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:38.348 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:38 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:42:38.348 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:38 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:42:38.348 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:38 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:38.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:38 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:38.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:38 smithi146 ceph-mon[26954]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:38.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:38 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:42:38.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:38 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:42:38.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:38 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:39.295 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:39 smithi032 ceph-mon[24793]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:39.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:39 smithi146 ceph-mon[26954]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:41.037 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:40 smithi032 ceph-mon[24793]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:41.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:40 smithi146 ceph-mon[26954]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:41.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi032 to 6434M 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:41.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:42.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:42.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:42.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:42.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi032 to 6434M 2024-03-24T11:42:42.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:42.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:42.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:42.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:42.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:42 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1384474436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4a7326a-5ac8-4e97-bff3-a4291969db49"}]: dispatch 2024-03-24T11:42:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:42 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1384474436' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4a7326a-5ac8-4e97-bff3-a4291969db49"}]': finished 2024-03-24T11:42:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:42 smithi146 ceph-mon[26954]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:42:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:42 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1704857912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:42:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:42 smithi146 ceph-mon[26954]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:43.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:42 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1384474436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4a7326a-5ac8-4e97-bff3-a4291969db49"}]: dispatch 2024-03-24T11:42:43.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:42 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1384474436' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4a7326a-5ac8-4e97-bff3-a4291969db49"}]': finished 2024-03-24T11:42:43.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:42 smithi032 ceph-mon[24793]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:42:43.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:43.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:42 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1704857912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:42:43.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:42 smithi032 ceph-mon[24793]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:45.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:44 smithi146 ceph-mon[26954]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:45.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:44 smithi032 ceph-mon[24793]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:46.888 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:46 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:46.888 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:46 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:46 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:46 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:47.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:47 smithi032 ceph-mon[24793]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:47 smithi146 ceph-mon[26954]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:48.941 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:48 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:42:48.942 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:48 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:49.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:48 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:42:49.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:48 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:49.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:49 smithi032 ceph-mon[24793]: Deploying daemon osd.2 on smithi032 2024-03-24T11:42:49.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:49 smithi032 ceph-mon[24793]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:50.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:49 smithi146 ceph-mon[26954]: Deploying daemon osd.2 on smithi032 2024-03-24T11:42:50.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:49 smithi146 ceph-mon[26954]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:51.133 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:50 smithi032 ceph-mon[24793]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:51.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:50 smithi146 ceph-mon[26954]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:53.040 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:52 smithi032 ceph-mon[24793]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:53.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:52 smithi146 ceph-mon[26954]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:54.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:54 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:54.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:54 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:54.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:54 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:54.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:54 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:54.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:54 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:54.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:54 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:54.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:54 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:54.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:54 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:55.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:55 smithi032 ceph-mon[24793]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:55.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:55 smithi146 ceph-mon[26954]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:56.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:56 smithi032 ceph-mon[24793]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:42:56.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:56 smithi146 ceph-mon[26954]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:42:56.851 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 2 on host 'smithi032' 2024-03-24T11:42:57.495 DEBUG:teuthology.orchestra.run.smithi032:osd.2> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:42:57.497 INFO:tasks.cephadm:Deploying osd.3 on smithi032 with /dev/vg_nvme/lv_1... 2024-03-24T11:42:57.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:42:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:42:57.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:42:58.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:58 smithi032 ceph-mon[24793]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:42:58.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:58 smithi032 ceph-mon[24793]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T11:42:58.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:58 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:58.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:58 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:58.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:58 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:58.888 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:58 smithi146 ceph-mon[26954]: from='osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:42:58.888 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:58 smithi146 ceph-mon[26954]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T11:42:58.888 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:58 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:58.888 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:58 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:58.888 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:58 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:59.575 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:42:59.576 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:42:59.576 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-03-24T11:42:59.576 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-03-24T11:42:59.576 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00970638 s, 1.1 GB/s 2024-03-24T11:42:59.576 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-03-24T11:42:59.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277] boot 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:42:59.706 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:42:59 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:42:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:42:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: osd.2 [v2:172.21.15.32:6818/3671793277,v1:172.21.15.32:6819/3671793277] boot 2024-03-24T11:42:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:42:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:42:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:42:59.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T11:42:59.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T11:42:59.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:42:59.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:42:59 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:00.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi032:vg_nvme/lv_1 2024-03-24T11:43:01.240 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:43:00 smithi032 sudo[44246]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:43:01.240 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:43:00 smithi032 sudo[44246]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:01.240 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:43:00 smithi032 sudo[44246]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:01.240 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44278]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:43:01.515 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44278]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:01.515 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44278]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:01.515 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:01.515 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:01 smithi032 ceph-mon[24793]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:43:01.515 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:01 smithi032 ceph-mon[24793]: pgmap v71: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:01.515 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:01.773 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44281]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:43:01.773 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44281]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:01.773 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44281]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:01.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:01 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:01.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:01 smithi146 ceph-mon[26954]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:43:01.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:01 smithi146 ceph-mon[26954]: pgmap v71: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:01.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:01 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:02.046 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44309]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:43:02.046 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44309]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:02.046 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:43:01 smithi032 sudo[44309]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:02.313 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44341]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:43:02.314 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44341]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:02.314 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44341]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:02.571 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44365]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:43:02.571 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44365]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:02.571 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44365]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:02.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44368]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:43:02.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44368]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:02.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44368]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:03.105 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:02 smithi146 ceph-mon[26954]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:43:03.105 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:02 smithi146 ceph-mon[26954]: mgrmap e19: smithi032.xjdgbw(active, since 2m), standbys: smithi146.tfohfl 2024-03-24T11:43:03.105 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:02 smithi146 ceph-mon[26954]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:43:03.105 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:02 smithi146 ceph-mon[26954]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:43:03.105 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 sudo[29271]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:43:03.105 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 sudo[29271]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:03.105 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 sudo[29271]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:03.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44389]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:43:03.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44389]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:03.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 sudo[44389]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:03.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 ceph-mon[24793]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:43:03.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 ceph-mon[24793]: mgrmap e19: smithi032.xjdgbw(active, since 2m), standbys: smithi146.tfohfl 2024-03-24T11:43:03.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:43:03.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:02 smithi032 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:43:03.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 sudo[29274]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:43:03.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 sudo[29274]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:43:03.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 sudo[29274]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:43:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:43:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:43:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:43:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:43:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:43:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:43:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:43:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:43:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:03 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:04.158 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:43:04.159 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:03 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:04 smithi146 ceph-mon[26954]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:05.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:04 smithi032 ceph-mon[24793]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: Detected new or changed devices on smithi032 2024-03-24T11:43:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi032 to 4289M 2024-03-24T11:43:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/120846144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e87c2b-4340-41e4-9334-398c0931f237"}]: dispatch 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/120846144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e87c2b-4340-41e4-9334-398c0931f237"}]': finished 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:43:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:07 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: Detected new or changed devices on smithi032 2024-03-24T11:43:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:07.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi032 to 4289M 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/120846144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e87c2b-4340-41e4-9334-398c0931f237"}]: dispatch 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/120846144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e87c2b-4340-41e4-9334-398c0931f237"}]': finished 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:43:07.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:07 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:08.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:08 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1956006486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:43:08.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:08 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1956006486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:43:09.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:09 smithi146 ceph-mon[26954]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:09.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:09 smithi032 ceph-mon[24793]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:10.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:10 smithi032 ceph-mon[24793]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:10 smithi146 ceph-mon[26954]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:13.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:12 smithi032 ceph-mon[24793]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:12 smithi146 ceph-mon[26954]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:14.272 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:13 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:43:14.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:13 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:14.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:13 smithi032 ceph-mon[24793]: Deploying daemon osd.3 on smithi032 2024-03-24T11:43:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:13 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:43:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:13 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:13 smithi146 ceph-mon[26954]: Deploying daemon osd.3 on smithi032 2024-03-24T11:43:15.138 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:14 smithi032 ceph-mon[24793]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:15.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:14 smithi146 ceph-mon[26954]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:16.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:16 smithi032 ceph-mon[24793]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:17.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:16 smithi146 ceph-mon[26954]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:19 smithi146 ceph-mon[26954]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:19.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:19.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:19.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:19.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:19.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:19 smithi032 ceph-mon[24793]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:20.791 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:20 smithi032 ceph-mon[24793]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:21.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:20 smithi146 ceph-mon[26954]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:21.643 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 3 on host 'smithi032' 2024-03-24T11:43:22.200 DEBUG:teuthology.orchestra.run.smithi032:osd.3> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T11:43:22.203 INFO:tasks.cephadm:Deploying osd.4 on smithi146 with /dev/vg_nvme/lv_4... 2024-03-24T11:43:22.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:43:22.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:22 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:22.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:22 smithi032 ceph-mon[24793]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:43:22.828 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:22 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:22.828 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:22 smithi146 ceph-mon[26954]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:43:23.748 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:23 smithi146 ceph-mon[26954]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:43:23.748 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:23 smithi146 ceph-mon[26954]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:43:23.748 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:23 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:23.748 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:23 smithi146 ceph-mon[26954]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:43:23.748 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:23 smithi146 ceph-mon[26954]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:23.748 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:23 smithi146 ceph-mon[26954]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:43:23.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-mon[24793]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:43:23.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-mon[24793]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:43:23.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:23.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-mon[24793]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:43:23.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-mon[24793]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:23.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-mon[24793]: from='osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-03-24T11:43:23.953 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[46208]: 2024-03-24T11:43:23.648+0000 7f7972025700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T11:43:23.954 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:43:23 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[46208]: 2024-03-24T11:43:23.653+0000 7f796c9bc700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:43:24.319 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:43:24.319 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:43:24.319 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-24T11:43:24.319 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-24T11:43:24.319 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011745 s, 893 MB/s 2024-03-24T11:43:24.319 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-24T11:43:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:24 smithi146 ceph-mon[26954]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:43:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:24.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2024-03-24T11:43:24.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:24 smithi032 ceph-mon[24793]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:43:24.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:24.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:24.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:25.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:25 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:43:25.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:25 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:43:25.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:25 smithi032 ceph-mon[24793]: osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643] boot 2024-03-24T11:43:25.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:25 smithi032 ceph-mon[24793]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:43:25.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:25.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:25 smithi032 ceph-mon[24793]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:25.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:25 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:43:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:25 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:43:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:25 smithi146 ceph-mon[26954]: osd.3 [v2:172.21.15.32:6826/153369643,v1:172.21.15.32:6827/153369643] boot 2024-03-24T11:43:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:25 smithi146 ceph-mon[26954]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:43:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:43:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:25 smithi146 ceph-mon[26954]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:43:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:26.847 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:26 smithi146 ceph-mon[26954]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:43:26.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:26 smithi032 ceph-mon[24793]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:43:27.827 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:27 smithi146 ceph-mon[26954]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:27.828 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:27 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:43:27.828 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:27 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:43:27.828 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:27 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:27.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:27 smithi032 ceph-mon[24793]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:27.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:27 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:43:27.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:27 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:43:27.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:27 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:28.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:28 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:28 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:29.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: Detected new or changed devices on smithi032 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi032 to 3217M 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1128703035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8a3113-710d-4ac3-a14d-524ca4df7822"}]: dispatch 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8a3113-710d-4ac3-a14d-524ca4df7822"}]: dispatch 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f8a3113-710d-4ac3-a14d-524ca4df7822"}]': finished 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:29.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:29 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1246960663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:43:30.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: Detected new or changed devices on smithi032 2024-03-24T11:43:30.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi032 to 3217M 2024-03-24T11:43:30.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:30.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/1128703035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8a3113-710d-4ac3-a14d-524ca4df7822"}]: dispatch 2024-03-24T11:43:30.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8a3113-710d-4ac3-a14d-524ca4df7822"}]: dispatch 2024-03-24T11:43:30.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f8a3113-710d-4ac3-a14d-524ca4df7822"}]': finished 2024-03-24T11:43:30.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:43:30.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:30.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:29 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/1246960663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:43:31.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:30 smithi146 ceph-mon[26954]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:30 smithi032 ceph-mon[24793]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:32 smithi146 ceph-mon[26954]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:33.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:32 smithi032 ceph-mon[24793]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:34 smithi146 ceph-mon[26954]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:35.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:34 smithi032 ceph-mon[24793]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:35.865 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:35 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:43:35.865 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:35 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:35.865 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:35 smithi146 ceph-mon[26954]: Deploying daemon osd.4 on smithi146 2024-03-24T11:43:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:35 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:43:36.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:35 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:36.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:35 smithi032 ceph-mon[24793]: Deploying daemon osd.4 on smithi146 2024-03-24T11:43:37.076 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:36 smithi032 ceph-mon[24793]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:37.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:36 smithi146 ceph-mon[26954]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:39.124 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:38 smithi146 ceph-mon[26954]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:39.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:38 smithi032 ceph-mon[24793]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:41.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:41.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:41.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:41.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:41.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:41 smithi146 ceph-mon[26954]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:41 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:41 smithi032 ceph-mon[24793]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:42 smithi146 ceph-mon[26954]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:43.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:42 smithi032 ceph-mon[24793]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:43.845 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 4 on host 'smithi146' 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='osd.4 [v2:172.21.15.146:6800/3324113540,v1:172.21.15.146:6801/3324113540]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:44.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='osd.4 [v2:172.21.15.146:6800/3324113540,v1:172.21.15.146:6801/3324113540]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:44.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:44.507 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T11:43:44.509 INFO:tasks.cephadm:Deploying osd.5 on smithi146 with /dev/vg_nvme/lv_3... 2024-03-24T11:43:44.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:43:45.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-mon[26954]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:43:45.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-mon[26954]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:43:45.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:45.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-mon[26954]: from='osd.4 [v2:172.21.15.146:6800/3324113540,v1:172.21.15.146:6801/3324113540]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:43:45.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-mon[26954]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:43:45.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-mon[26954]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:45.386 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[31212]: 2024-03-24T11:43:44.992+0000 7fc2d87ca700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T11:43:45.386 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:43:44 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[31212]: 2024-03-24T11:43:44.996+0000 7fc2d3161700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:43:45.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:44 smithi032 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:43:45.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:44 smithi032 ceph-mon[24793]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:43:45.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:45.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:44 smithi032 ceph-mon[24793]: from='osd.4 [v2:172.21.15.146:6800/3324113540,v1:172.21.15.146:6801/3324113540]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:43:45.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:44 smithi032 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:43:45.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:44 smithi032 ceph-mon[24793]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:43:46.301 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:45 smithi146 ceph-mon[26954]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:43:46.301 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:45 smithi146 ceph-mon[26954]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:43:46.301 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:45 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:46.301 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:45 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:46.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:45 smithi032 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:43:46.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:45 smithi032 ceph-mon[24793]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:43:46.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:45 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:46.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:45 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:46 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:43:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:46 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:43:47.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:46 smithi146 ceph-mon[26954]: osd.4 [v2:172.21.15.146:6800/3324113540,v1:172.21.15.146:6801/3324113540] boot 2024-03-24T11:43:47.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:46 smithi146 ceph-mon[26954]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:43:47.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:46 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:47.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:46 smithi146 ceph-mon[26954]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:43:47.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:46 smithi146 ceph-mon[26954]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:43:47.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:46 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:43:47.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:46 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:43:47.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:46 smithi032 ceph-mon[24793]: osd.4 [v2:172.21.15.146:6800/3324113540,v1:172.21.15.146:6801/3324113540] boot 2024-03-24T11:43:47.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:46 smithi032 ceph-mon[24793]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:43:47.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:46 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:43:47.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:46 smithi032 ceph-mon[24793]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:43:47.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:46 smithi032 ceph-mon[24793]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:43:48.680 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:43:48.681 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:43:48.681 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-24T11:43:48.681 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-24T11:43:48.681 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126886 s, 826 MB/s 2024-03-24T11:43:48.681 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 2024-03-24T11:43:48.681 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-24T11:43:48.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:48 smithi146 ceph-mon[26954]: Detected new or changed devices on smithi146 2024-03-24T11:43:48.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:48 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:48.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:48 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:48.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:48 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi146 to 15936M 2024-03-24T11:43:48.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:48 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:48.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:48 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:48.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:48 smithi146 ceph-mon[26954]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:43:48.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:48 smithi032 ceph-mon[24793]: Detected new or changed devices on smithi146 2024-03-24T11:43:48.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:48 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:48.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:48 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:43:48.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:48 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi146 to 15936M 2024-03-24T11:43:48.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:48 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:48.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:48 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:43:48.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:48 smithi032 ceph-mon[24793]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:43:49.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2024-03-24T11:43:49.802 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:49 smithi146 ceph-mon[26954]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:43:49.802 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:49 smithi146 ceph-mon[26954]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:43:49.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:49 smithi032 ceph-mon[24793]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:43:49.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:49 smithi032 ceph-mon[24793]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:43:51.354 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:50 smithi146 ceph-mon[26954]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:43:51.354 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:50 smithi146 ceph-mon[26954]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-24T11:43:51.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:50 smithi032 ceph-mon[24793]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:43:51.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:50 smithi032 ceph-mon[24793]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-24T11:43:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:51 smithi146 ceph-mon[26954]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:51 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:43:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:51 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:43:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:51 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:52.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:51 smithi032 ceph-mon[24793]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:52.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:51 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:43:52.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:51 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:43:52.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:51 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:53.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:52 smithi146 ceph-mon[26954]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-24T11:43:53.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:52 smithi032 ceph-mon[24793]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-24T11:43:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:53 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/1816899791' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22947993-6c1a-44b3-a06f-ac0c0dc4f1c7"}]: dispatch 2024-03-24T11:43:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:53 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22947993-6c1a-44b3-a06f-ac0c0dc4f1c7"}]: dispatch 2024-03-24T11:43:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:53 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22947993-6c1a-44b3-a06f-ac0c0dc4f1c7"}]': finished 2024-03-24T11:43:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:53 smithi146 ceph-mon[26954]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:43:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:53 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:43:54.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:53 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1816899791' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22947993-6c1a-44b3-a06f-ac0c0dc4f1c7"}]: dispatch 2024-03-24T11:43:54.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:53 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22947993-6c1a-44b3-a06f-ac0c0dc4f1c7"}]: dispatch 2024-03-24T11:43:54.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:53 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22947993-6c1a-44b3-a06f-ac0c0dc4f1c7"}]': finished 2024-03-24T11:43:54.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:53 smithi032 ceph-mon[24793]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:43:54.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:53 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:43:55.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:54 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/2045204430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:43:55.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:54 smithi146 ceph-mon[26954]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:43:55.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:54 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/2045204430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:43:55.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:54 smithi032 ceph-mon[24793]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:43:57.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:56 smithi032 ceph-mon[24793]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T11:43:57.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:43:57.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:43:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:56 smithi146 ceph-mon[26954]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T11:43:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:43:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:43:58.928 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:43:58 smithi146 ceph-mon[26954]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:43:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:43:58 smithi032 ceph-mon[24793]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:00.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:00 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:44:00.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:00 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:00.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:00 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:44:00.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:00 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:01.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:01 smithi146 ceph-mon[26954]: Deploying daemon osd.5 on smithi146 2024-03-24T11:44:01.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:01 smithi146 ceph-mon[26954]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:01.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:01 smithi032 ceph-mon[24793]: Deploying daemon osd.5 on smithi146 2024-03-24T11:44:01.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:01 smithi032 ceph-mon[24793]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:02 smithi146 ceph-mon[26954]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:03.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:02 smithi032 ceph-mon[24793]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:05.127 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:04 smithi146 ceph-mon[26954]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:04 smithi032 ceph-mon[24793]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:06.574 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:06.575 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:06.575 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:06.575 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:06 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:06.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:06.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:06.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:06.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:06 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:07 smithi032 ceph-mon[24793]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:07.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:07 smithi146 ceph-mon[26954]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:08.585 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:08 smithi146 ceph-mon[26954]: from='osd.5 [v2:172.21.15.146:6808/1567683755,v1:172.21.15.146:6809/1567683755]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:44:08.585 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:08 smithi146 ceph-mon[26954]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:44:08.617 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 5 on host 'smithi146' 2024-03-24T11:44:08.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:08 smithi032 ceph-mon[24793]: from='osd.5 [v2:172.21.15.146:6808/1567683755,v1:172.21.15.146:6809/1567683755]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:44:08.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:08 smithi032 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:44:09.281 DEBUG:teuthology.orchestra.run.smithi146:osd.5> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T11:44:09.284 INFO:tasks.cephadm:Deploying osd.6 on smithi146 with /dev/vg_nvme/lv_2... 2024-03-24T11:44:09.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:44:09.577 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[34191]: 2024-03-24T11:44:09.436+0000 7ff7135f9700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T11:44:09.577 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[34191]: 2024-03-24T11:44:09.443+0000 7ff70df90700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:44:09.577 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-mon[26954]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:44:09.577 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-mon[26954]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:44:09.578 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:09.578 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-mon[26954]: from='osd.5 [v2:172.21.15.146:6808/1567683755,v1:172.21.15.146:6809/1567683755]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:09.578 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-mon[26954]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:09.578 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:09.578 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:09 smithi146 ceph-mon[26954]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:09.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:09 smithi032 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:44:09.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:09 smithi032 ceph-mon[24793]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:44:09.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:09 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:09.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:09 smithi032 ceph-mon[24793]: from='osd.5 [v2:172.21.15.146:6808/1567683755,v1:172.21.15.146:6809/1567683755]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:09.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:09 smithi032 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:09.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:09 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:09.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:09 smithi032 ceph-mon[24793]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:10.626 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:10 smithi146 ceph-mon[26954]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:44:10.627 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:10 smithi146 ceph-mon[26954]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:44:10.627 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:10 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:10.627 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:10 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:10.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:10 smithi032 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:44:10.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:10 smithi032 ceph-mon[24793]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:44:10.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:10.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:10 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:44:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:44:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: osd.5 [v2:172.21.15.146:6808/1567683755,v1:172.21.15.146:6809/1567683755] boot 2024-03-24T11:44:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:44:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:11.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:11 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: osd.5 [v2:172.21.15.146:6808/1567683755,v1:172.21.15.146:6809/1567683755] boot 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:44:11.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:11 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:11.817 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:44:11.817 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:44:11.817 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-24T11:44:11.817 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-24T11:44:11.817 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136858 s, 766 MB/s 2024-03-24T11:44:11.817 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-24T11:44:12.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:12 smithi032 ceph-mon[24793]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:44:12.868 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:12 smithi146 ceph-mon[26954]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:44:12.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2024-03-24T11:44:13.618 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:13 smithi146 ceph-mon[26954]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:44:13.619 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:13 smithi146 ceph-mon[26954]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:13.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:13 smithi032 ceph-mon[24793]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:44:13.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:13 smithi032 ceph-mon[24793]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:14.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:14 smithi146 ceph-mon[26954]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:44:14.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:14 smithi032 ceph-mon[24793]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:44:15.717 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: pgmap v130: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:15.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:15 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: pgmap v130: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:15.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:15.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:15 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:16.623 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:16 smithi146 ceph-mon[26954]: Detected new or changed devices on smithi146 2024-03-24T11:44:16.623 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:16 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi146 to 7968M 2024-03-24T11:44:16.624 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:16 smithi146 ceph-mon[26954]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-24T11:44:16.624 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:16 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:44:16.624 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:16 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:44:16.624 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:16 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:16.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:16 smithi032 ceph-mon[24793]: Detected new or changed devices on smithi146 2024-03-24T11:44:16.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:16 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi146 to 7968M 2024-03-24T11:44:16.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:16 smithi032 ceph-mon[24793]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-24T11:44:16.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:16 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:44:16.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:16 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:44:16.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:16 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:17.788 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:17 smithi146 ceph-mon[26954]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:17.788 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:17 smithi146 ceph-mon[26954]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:17.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:17 smithi032 ceph-mon[24793]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:17.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:17 smithi032 ceph-mon[24793]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:18.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:18 smithi146 ceph-mon[26954]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-24T11:44:18.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:18 smithi146 ceph-mon[26954]: Cluster is now healthy 2024-03-24T11:44:18.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:18 smithi032 ceph-mon[24793]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-24T11:44:18.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:18 smithi032 ceph-mon[24793]: Cluster is now healthy 2024-03-24T11:44:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:19 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/1999096213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "badcd75d-dd8b-4c2a-bdfb-c46b056654bb"}]: dispatch 2024-03-24T11:44:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:19 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "badcd75d-dd8b-4c2a-bdfb-c46b056654bb"}]: dispatch 2024-03-24T11:44:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:19 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "badcd75d-dd8b-4c2a-bdfb-c46b056654bb"}]': finished 2024-03-24T11:44:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:19 smithi146 ceph-mon[26954]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:44:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:19 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:19 smithi146 ceph-mon[26954]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:19.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:19 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/1478413056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:44:19.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:19 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1999096213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "badcd75d-dd8b-4c2a-bdfb-c46b056654bb"}]: dispatch 2024-03-24T11:44:19.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:19 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "badcd75d-dd8b-4c2a-bdfb-c46b056654bb"}]: dispatch 2024-03-24T11:44:19.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:19 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "badcd75d-dd8b-4c2a-bdfb-c46b056654bb"}]': finished 2024-03-24T11:44:19.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:19 smithi032 ceph-mon[24793]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:44:19.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:19 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:19.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:19 smithi032 ceph-mon[24793]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:19.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:19 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1478413056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:44:21.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:20 smithi146 ceph-mon[26954]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-24T11:44:21.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:20 smithi032 ceph-mon[24793]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-24T11:44:23.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:22 smithi146 ceph-mon[26954]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-24T11:44:23.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:22 smithi032 ceph-mon[24793]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-24T11:44:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:24 smithi146 ceph-mon[26954]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:44:25.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:24 smithi032 ceph-mon[24793]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:44:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:44:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:25.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:25 smithi146 ceph-mon[26954]: Deploying daemon osd.6 on smithi146 2024-03-24T11:44:26.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:44:26.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:26.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:25 smithi032 ceph-mon[24793]: Deploying daemon osd.6 on smithi146 2024-03-24T11:44:26.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:26 smithi146 ceph-mon[26954]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:27.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:26 smithi032 ceph-mon[24793]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:28 smithi146 ceph-mon[26954]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:29.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:28 smithi032 ceph-mon[24793]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:31 smithi146 ceph-mon[26954]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:31.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:31.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:31.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:31.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:31.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:31 smithi032 ceph-mon[24793]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:33.319 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:32 smithi146 ceph-mon[26954]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:33.349 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 6 on host 'smithi146' 2024-03-24T11:44:33.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:32 smithi032 ceph-mon[24793]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:34.112 DEBUG:teuthology.orchestra.run.smithi146:osd.6> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T11:44:34.115 INFO:tasks.cephadm:Deploying osd.7 on smithi146 with /dev/vg_nvme/lv_1... 2024-03-24T11:44:34.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:44:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:33 smithi146 ceph-mon[26954]: from='osd.6 [v2:172.21.15.146:6816/1086572132,v1:172.21.15.146:6817/1086572132]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:44:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:33 smithi146 ceph-mon[26954]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:44:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:33 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:34.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:33 smithi032 ceph-mon[24793]: from='osd.6 [v2:172.21.15.146:6816/1086572132,v1:172.21.15.146:6817/1086572132]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:44:34.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:33 smithi032 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:44:34.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:33 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:35.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:34 smithi146 ceph-mon[26954]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:44:35.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:34 smithi146 ceph-mon[26954]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:44:35.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:34 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:35.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:34 smithi146 ceph-mon[26954]: from='osd.6 [v2:172.21.15.146:6816/1086572132,v1:172.21.15.146:6817/1086572132]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:35.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:34 smithi146 ceph-mon[26954]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:35.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:34 smithi146 ceph-mon[26954]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:35.346 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:44:35 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[37180]: 2024-03-24T11:44:35.003+0000 7fb8b18dc700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T11:44:35.347 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:44:35 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[37180]: 2024-03-24T11:44:35.009+0000 7fb8ac273700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:44:35.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:34 smithi032 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:44:35.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:34 smithi032 ceph-mon[24793]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:44:35.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:34 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:35.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:34 smithi032 ceph-mon[24793]: from='osd.6 [v2:172.21.15.146:6816/1086572132,v1:172.21.15.146:6817/1086572132]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:35.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:34 smithi032 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:44:35.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:34 smithi032 ceph-mon[24793]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:44:36.300 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:36 smithi146 ceph-mon[26954]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:44:36.300 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:36 smithi146 ceph-mon[26954]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:44:36.300 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:36 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:36.300 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:36 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:36.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:36 smithi032 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:44:36.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:36 smithi032 ceph-mon[24793]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:44:36.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:36 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:36.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:36 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:36.797 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:44:36.798 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:44:36.798 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-24T11:44:36.798 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-24T11:44:36.798 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137267 s, 764 MB/s 2024-03-24T11:44:36.798 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-24T11:44:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:44:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:44:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: osd.6 [v2:172.21.15.146:6816/1086572132,v1:172.21.15.146:6817/1086572132] boot 2024-03-24T11:44:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:44:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:37.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:37.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:37 smithi146 ceph-mon[26954]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: osd.6 [v2:172.21.15.146:6816/1086572132,v1:172.21.15.146:6817/1086572132] boot 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:37.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:37 smithi032 ceph-mon[24793]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:44:37.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_1 2024-03-24T11:44:39.050 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:38 smithi146 ceph-mon[26954]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:44:39.051 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:38 smithi146 ceph-mon[26954]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:39.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:38 smithi032 ceph-mon[24793]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:44:39.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:38 smithi032 ceph-mon[24793]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:40.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:39 smithi146 ceph-mon[26954]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:44:40.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:39 smithi032 ceph-mon[24793]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:44:41.265 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:41 smithi146 ceph-mon[26954]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:44:41.265 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:41 smithi146 ceph-mon[26954]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:41.265 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:44:41.265 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:44:41.266 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:41 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:41.266 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:41 smithi146 ceph-mon[26954]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:40 smithi032 ceph-mon[24793]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:44:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:40 smithi032 ceph-mon[24793]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:40 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:44:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:40 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:44:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:40 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:41.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:40 smithi032 ceph-mon[24793]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: Detected new or changed devices on smithi146 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi146 to 5312M 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:43.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:43.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:43.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:43 smithi146 ceph-mon[26954]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: Detected new or changed devices on smithi146 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi146 to 5312M 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:43.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:43.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:43.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:43 smithi032 ceph-mon[24793]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-24T11:44:44.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:44 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/1545621886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39223732-7eb6-403f-bf5a-3a74f6621ec9"}]: dispatch 2024-03-24T11:44:44.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:44 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39223732-7eb6-403f-bf5a-3a74f6621ec9"}]: dispatch 2024-03-24T11:44:44.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:44 smithi146 ceph-mon[26954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39223732-7eb6-403f-bf5a-3a74f6621ec9"}]': finished 2024-03-24T11:44:44.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:44 smithi146 ceph-mon[26954]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:44:44.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:44:44.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:44 smithi146 ceph-mon[26954]: from='client.? 172.21.15.146:0/3202189212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:44:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:44 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/1545621886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39223732-7eb6-403f-bf5a-3a74f6621ec9"}]: dispatch 2024-03-24T11:44:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:44 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39223732-7eb6-403f-bf5a-3a74f6621ec9"}]: dispatch 2024-03-24T11:44:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:44 smithi032 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39223732-7eb6-403f-bf5a-3a74f6621ec9"}]': finished 2024-03-24T11:44:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:44 smithi032 ceph-mon[24793]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:44:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:44:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:44 smithi032 ceph-mon[24793]: from='client.? 172.21.15.146:0/3202189212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:44:45.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:45 smithi146 ceph-mon[26954]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:44:45.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:45 smithi032 ceph-mon[24793]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:44:47.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:46 smithi032 ceph-mon[24793]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T11:44:47.126 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:46 smithi146 ceph-mon[26954]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T11:44:49.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:48 smithi146 ceph-mon[26954]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T11:44:49.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:48 smithi032 ceph-mon[24793]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T11:44:49.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:49 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:44:49.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:49 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:49.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:49 smithi146 ceph-mon[26954]: Deploying daemon osd.7 on smithi146 2024-03-24T11:44:50.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:49 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:44:50.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:49 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:50.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:49 smithi032 ceph-mon[24793]: Deploying daemon osd.7 on smithi146 2024-03-24T11:44:51.047 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:50 smithi146 ceph-mon[26954]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:44:51.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:50 smithi032 ceph-mon[24793]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:44:53.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:52 smithi146 ceph-mon[26954]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:53.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:52 smithi032 ceph-mon[24793]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:55.055 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:54 smithi146 ceph-mon[26954]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:55.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:54 smithi032 ceph-mon[24793]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:56.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:56.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:56.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:56.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:56.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:56.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:56.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:56.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:57.282 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:57 smithi146 ceph-mon[26954]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:57.282 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:44:57.282 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:44:57.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:57 smithi032 ceph-mon[24793]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:57.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:44:57.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:44:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 7 on host 'smithi146' 2024-03-24T11:44:58.659 DEBUG:teuthology.orchestra.run.smithi146:osd.7> sudo journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T11:44:58.662 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-24T11:44:58.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:44:59.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:59 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:59.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:59 smithi032 ceph-mon[24793]: from='osd.7 [v2:172.21.15.146:6824/2888776911,v1:172.21.15.146:6825/2888776911]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:44:59.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:59 smithi032 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:44:59.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:44:59 smithi032 ceph-mon[24793]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:44:59.321 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:59 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:44:59.321 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:59 smithi146 ceph-mon[26954]: from='osd.7 [v2:172.21.15.146:6824/2888776911,v1:172.21.15.146:6825/2888776911]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:44:59.321 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:59 smithi146 ceph-mon[26954]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:44:59.321 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:44:59 smithi146 ceph-mon[26954]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:45:00.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:00 smithi032 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:45:00.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:00 smithi032 ceph-mon[24793]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:45:00.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:00 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:00.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:00 smithi032 ceph-mon[24793]: from='osd.7 [v2:172.21.15.146:6824/2888776911,v1:172.21.15.146:6825/2888776911]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:45:00.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:00 smithi032 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:45:00.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:00 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:00.360 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-mon[26954]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:45:00.361 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-mon[26954]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:45:00.361 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:00.361 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-mon[26954]: from='osd.7 [v2:172.21.15.146:6824/2888776911,v1:172.21.15.146:6825/2888776911]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:45:00.361 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-mon[26954]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:45:00.361 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:00.361 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[40263]: 2024-03-24T11:45:00.007+0000 7facc4726700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T11:45:00.361 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:45:00 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[40263]: 2024-03-24T11:45:00.015+0000 7facbe8bc700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:45:00.702 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:01.345 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:01 smithi146 ceph-mon[26954]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:45:01.345 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:01 smithi146 ceph-mon[26954]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:45:01.345 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:01 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:01.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:01 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/81034298' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:45:01.346 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:01 smithi146 ceph-mon[26954]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:45:01.377 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:01 smithi032 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-24T11:45:01.377 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:01 smithi032 ceph-mon[24793]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:45:01.377 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:01 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:01.377 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:01 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/81034298' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:45:01.378 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:01 smithi032 ceph-mon[24793]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:45:01.401 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1711280675,"num_in_osds":8,"osd_in_since":1711280683,"num_remapped_pgs":0} 2024-03-24T11:45:02.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:02 smithi146 ceph-mon[26954]: purged_snaps scrub starts 2024-03-24T11:45:02.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:02 smithi146 ceph-mon[26954]: purged_snaps scrub ok 2024-03-24T11:45:02.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:02 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:02.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:02 smithi146 ceph-mon[26954]: osd.7 [v2:172.21.15.146:6824/2888776911,v1:172.21.15.146:6825/2888776911] boot 2024-03-24T11:45:02.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:02 smithi146 ceph-mon[26954]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:45:02.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:02 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:02.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:02 smithi146 ceph-mon[26954]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:45:02.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:45:02.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:02 smithi032 ceph-mon[24793]: purged_snaps scrub starts 2024-03-24T11:45:02.429 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:02 smithi032 ceph-mon[24793]: purged_snaps scrub ok 2024-03-24T11:45:02.429 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:02 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:02.429 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:02 smithi032 ceph-mon[24793]: osd.7 [v2:172.21.15.146:6824/2888776911,v1:172.21.15.146:6825/2888776911] boot 2024-03-24T11:45:02.429 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:02 smithi032 ceph-mon[24793]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:45:02.429 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:02 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:45:02.429 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:02 smithi032 ceph-mon[24793]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:45:03.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:03 smithi146 ceph-mon[26954]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:03.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:03 smithi146 ceph-mon[26954]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:45:03.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:03 smithi032 ceph-mon[24793]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:03.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:03 smithi032 ceph-mon[24793]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: Detected new or changed devices on smithi146 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi146 to 3984M 2024-03-24T11:45:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:04.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:04.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:04.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:04 smithi032 ceph-mon[24793]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:45:04.231 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: Detected new or changed devices on smithi146 2024-03-24T11:45:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi146 to 3984M 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:45:04.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:04 smithi146 ceph-mon[26954]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:45:04.781 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1711280701,"num_in_osds":8,"osd_in_since":1711280683,"num_remapped_pgs":0} 2024-03-24T11:45:04.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:45:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:05.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:05 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3403271465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:45:05.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:05 smithi032 ceph-mon[24793]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:05.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:05 smithi032 ceph-mon[24793]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:45:05.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:05 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3403271465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:45:05.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:05 smithi146 ceph-mon[26954]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:05.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:05 smithi146 ceph-mon[26954]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:45:06.696 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:06.696 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","created":"2024-03-24T11:39:07.523110+0000","modified":"2024-03-24T11:45:04.991995+0000","last_up_change":"2024-03-24T11:45:01.005358+0000","last_in_change":"2024-03-24T11:44:43.246825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:42:59.003957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9ac184de-5c35-4015-b99e-75966907f527","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6803","nonce":249424652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6805","nonce":249424652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6809","nonce":249424652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6807","nonce":249424652}]},"public_addr":"172.21.15.32:6803/249424652","cluster_addr":"172.21.15.32:6805/249424652","heartbeat_back_addr":"172.21.15.32:6809/249424652","heartbeat_front_addr":"172.21.15.32:6807/249424652","state":["exists","up"]},{"osd":1,"uuid":"5d4ec7af-f4dd-4201-aae9-cb21009558dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6811","nonce":3261903682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6813","nonce":3261903682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6817","nonce":3261903682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6815","nonce":3261903682}]},"public_addr":"172.21.15.32:6811/3261903682","cluster_addr":"172.21.15.32:6813/3261903682","heartbeat_back_addr":"172.21.15.32:6817/3261903682","heartbeat_front_addr":"172.21.15.32:6815/3261903682","state":["exists","up"]},{"osd":2,"uuid":"e4a7326a-5ac8-4e97-bff3-a4291969db49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6819","nonce":3671793277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6821","nonce":3671793277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6825","nonce":3671793277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6823","nonce":3671793277}]},"public_addr":"172.21.15.32:6819/3671793277","cluster_addr":"172.21.15.32:6821/3671793277","heartbeat_back_addr":"172.21.15.32:6825/3671793277","heartbeat_front_addr":"172.21.15.32:6823/3671793277","state":["exists","up"]},{"osd":3,"uuid":"15e87c2b-4340-41e4-9334-398c0931f237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6827","nonce":153369643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6829","nonce":153369643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6833","nonce":153369643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6831","nonce":153369643}]},"public_addr":"172.21.15.32:6827/153369643","cluster_addr":"172.21.15.32:6829/153369643","heartbeat_back_addr":"172.21.15.32:6833/153369643","heartbeat_front_addr":"172.21.15.32:6831/153369643","state":["exists","up"]},{"osd":4,"uuid":"7f8a3113-710d-4ac3-a14d-524ca4df7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6801","nonce":3324113540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6803","nonce":3324113540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6807","nonce":3324113540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6805","nonce":3324113540}]},"public_addr":"172.21.15.146:6801/3324113540","cluster_addr":"172.21.15.146:6803/3324113540","heartbeat_back_addr":"172.21.15.146:6807/3324113540","heartbeat_front_addr":"172.21.15.146:6805/3324113540","state":["exists","up"]},{"osd":5,"uuid":"22947993-6c1a-44b3-a06f-ac0c0dc4f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6809","nonce":1567683755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6811","nonce":1567683755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6815","nonce":1567683755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6813","nonce":1567683755}]},"public_addr":"172.21.15.146:6809/1567683755","cluster_addr":"172.21.15.146:6811/1567683755","heartbeat_back_addr":"172.21.15.146:6815/1567683755","heartbeat_front_addr":"172.21.15.146:6813/1567683755","state":["exists","up"]},{"osd":6,"uuid":"badcd75d-dd8b-4c2a-bdfb-c46b056654bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6817","nonce":1086572132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6819","nonce":1086572132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6823","nonce":1086572132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6821","nonce":1086572132}]},"public_addr":"172.21.15.146:6817/1086572132","cluster_addr":"172.21.15.146:6819/1086572132","heartbeat_back_addr":"172.21.15.146:6823/1086572132","heartbeat_front_addr":"172.21.15.146:6821/1086572132","state":["exists","up"]},{"osd":7,"uuid":"39223732-7eb6-403f-bf5a-3a74f6621ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6825","nonce":2888776911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6827","nonce":2888776911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6831","nonce":2888776911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6829","nonce":2888776911}]},"public_addr":"172.21.15.146:6825/2888776911","cluster_addr":"172.21.15.146:6827/2888776911","heartbeat_back_addr":"172.21.15.146:6831/2888776911","heartbeat_front_addr":"172.21.15.146:6829/2888776911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:59.820459+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:31.517819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:57.327174+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:43:22.660002+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:43:44.238723+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:09.409360+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:34.224000+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:59.076527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2430898427":"2024-03-25T11:40:56.627105+0000","172.21.15.32:6801/810062755":"2024-03-25T11:40:56.627105+0000","172.21.15.32:6801/2255656955":"2024-03-25T11:40:05.741283+0000","172.21.15.32:6800/2255656955":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/3060386269":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/1248672124":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/153620913":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/3749458703":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/1364378637":"2024-03-25T11:39:36.628204+0000","172.21.15.32:0/3127983456":"2024-03-25T11:40:05.741283+0000","172.21.15.32:6800/1288508470":"2024-03-25T11:39:36.628204+0000","172.21.15.32:6800/810062755":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/2072521784":"2024-03-25T11:39:36.628204+0000","172.21.15.32:0/72936805":"2024-03-25T11:39:36.628204+0000","172.21.15.32:6801/1288508470":"2024-03-25T11:39:36.628204+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:45:06.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:06 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3098032988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:45:06.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:06 smithi032 ceph-mon[24793]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:06 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3098032988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:45:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:06 smithi146 ceph-mon[26954]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:07.331 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:42:59.003957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-24T11:45:07.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:45:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:08.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:08 smithi032 ceph-mon[24793]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:45:09.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:08 smithi146 ceph-mon[26954]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:45:09.386 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-24T11:45:09.909 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:45:09.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:45:10.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:09 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/461309447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:45:10.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:09 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/461309447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:45:11.036 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:10 smithi146 ceph-mon[26954]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T11:45:11.160 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:10 smithi032 ceph-mon[24793]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T11:45:11.896 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-24T11:45:11.897 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBHEgBmtWxCNRAAnTrLo2JEZDtB6eRjIqkT6w== 2024-03-24T11:45:12.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:11 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1944170785' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:45:12.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:11 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1944170785' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:45:12.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:11 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1944170785' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:45:12.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:11 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1944170785' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:45:12.611 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-24T11:45:12.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:45:12.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:45:12.655 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:45:12.655 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:45:12.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:45:12.912 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:13.279 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:12 smithi032 ceph-mon[24793]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T11:45:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:12 smithi146 ceph-mon[26954]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T11:45:14.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:15.131 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:14 smithi032 ceph-mon[24793]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:45:15.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:14 smithi146 ceph-mon[26954]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:45:15.519 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"active_gid":14202,"active_name":"smithi032.xjdgbw","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1527085039},{"type":"v1","addr":"172.21.15.32:6801","nonce":1527085039}]},"active_addr":"172.21.15.32:6801/1527085039","active_change":"2024-03-24T11:40:56.627400+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14220,"name":"smithi146.tfohfl","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.32:8443/","prometheus":"http://172.21.15.32:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1193236970}]},{"addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1133466584}]},{"addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":577954915}]},{"addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2044593301}]}]}} 2024-03-24T11:45:15.524 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:45:15.525 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:45:15.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:45:15.750 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:16.102 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:15 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/4095221598' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:45:16.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:15 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/4095221598' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:45:17.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:16 smithi032 ceph-mon[24793]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-03-24T11:45:17.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:16 smithi146 ceph-mon[26954]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-03-24T11:45:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:17.541 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","created":"2024-03-24T11:39:07.523110+0000","modified":"2024-03-24T11:45:04.991995+0000","last_up_change":"2024-03-24T11:45:01.005358+0000","last_in_change":"2024-03-24T11:44:43.246825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:42:59.003957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9ac184de-5c35-4015-b99e-75966907f527","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6803","nonce":249424652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6805","nonce":249424652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6809","nonce":249424652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6807","nonce":249424652}]},"public_addr":"172.21.15.32:6803/249424652","cluster_addr":"172.21.15.32:6805/249424652","heartbeat_back_addr":"172.21.15.32:6809/249424652","heartbeat_front_addr":"172.21.15.32:6807/249424652","state":["exists","up"]},{"osd":1,"uuid":"5d4ec7af-f4dd-4201-aae9-cb21009558dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6811","nonce":3261903682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6813","nonce":3261903682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6817","nonce":3261903682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6815","nonce":3261903682}]},"public_addr":"172.21.15.32:6811/3261903682","cluster_addr":"172.21.15.32:6813/3261903682","heartbeat_back_addr":"172.21.15.32:6817/3261903682","heartbeat_front_addr":"172.21.15.32:6815/3261903682","state":["exists","up"]},{"osd":2,"uuid":"e4a7326a-5ac8-4e97-bff3-a4291969db49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6819","nonce":3671793277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6821","nonce":3671793277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6825","nonce":3671793277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6823","nonce":3671793277}]},"public_addr":"172.21.15.32:6819/3671793277","cluster_addr":"172.21.15.32:6821/3671793277","heartbeat_back_addr":"172.21.15.32:6825/3671793277","heartbeat_front_addr":"172.21.15.32:6823/3671793277","state":["exists","up"]},{"osd":3,"uuid":"15e87c2b-4340-41e4-9334-398c0931f237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6827","nonce":153369643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6829","nonce":153369643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6833","nonce":153369643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6831","nonce":153369643}]},"public_addr":"172.21.15.32:6827/153369643","cluster_addr":"172.21.15.32:6829/153369643","heartbeat_back_addr":"172.21.15.32:6833/153369643","heartbeat_front_addr":"172.21.15.32:6831/153369643","state":["exists","up"]},{"osd":4,"uuid":"7f8a3113-710d-4ac3-a14d-524ca4df7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6801","nonce":3324113540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6803","nonce":3324113540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6807","nonce":3324113540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6805","nonce":3324113540}]},"public_addr":"172.21.15.146:6801/3324113540","cluster_addr":"172.21.15.146:6803/3324113540","heartbeat_back_addr":"172.21.15.146:6807/3324113540","heartbeat_front_addr":"172.21.15.146:6805/3324113540","state":["exists","up"]},{"osd":5,"uuid":"22947993-6c1a-44b3-a06f-ac0c0dc4f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6809","nonce":1567683755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6811","nonce":1567683755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6815","nonce":1567683755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6813","nonce":1567683755}]},"public_addr":"172.21.15.146:6809/1567683755","cluster_addr":"172.21.15.146:6811/1567683755","heartbeat_back_addr":"172.21.15.146:6815/1567683755","heartbeat_front_addr":"172.21.15.146:6813/1567683755","state":["exists","up"]},{"osd":6,"uuid":"badcd75d-dd8b-4c2a-bdfb-c46b056654bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6817","nonce":1086572132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6819","nonce":1086572132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6823","nonce":1086572132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6821","nonce":1086572132}]},"public_addr":"172.21.15.146:6817/1086572132","cluster_addr":"172.21.15.146:6819/1086572132","heartbeat_back_addr":"172.21.15.146:6823/1086572132","heartbeat_front_addr":"172.21.15.146:6821/1086572132","state":["exists","up"]},{"osd":7,"uuid":"39223732-7eb6-403f-bf5a-3a74f6621ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6825","nonce":2888776911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6827","nonce":2888776911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6831","nonce":2888776911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6829","nonce":2888776911}]},"public_addr":"172.21.15.146:6825/2888776911","cluster_addr":"172.21.15.146:6827/2888776911","heartbeat_back_addr":"172.21.15.146:6831/2888776911","heartbeat_front_addr":"172.21.15.146:6829/2888776911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:59.820459+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:31.517819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:57.327174+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:43:22.660002+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:43:44.238723+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:09.409360+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:34.224000+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:59.076527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2430898427":"2024-03-25T11:40:56.627105+0000","172.21.15.32:6801/810062755":"2024-03-25T11:40:56.627105+0000","172.21.15.32:6801/2255656955":"2024-03-25T11:40:05.741283+0000","172.21.15.32:6800/2255656955":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/3060386269":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/1248672124":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/153620913":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/3749458703":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/1364378637":"2024-03-25T11:39:36.628204+0000","172.21.15.32:0/3127983456":"2024-03-25T11:40:05.741283+0000","172.21.15.32:6800/1288508470":"2024-03-25T11:39:36.628204+0000","172.21.15.32:6800/810062755":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/2072521784":"2024-03-25T11:39:36.628204+0000","172.21.15.32:0/72936805":"2024-03-25T11:39:36.628204+0000","172.21.15.32:6801/1288508470":"2024-03-25T11:39:36.628204+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:45:18.149 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:45:18.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:45:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:18.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:17 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/853814661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:45:18.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:17 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/853814661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:45:19.152 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:18 smithi032 ceph-mon[24793]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-03-24T11:45:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:18 smithi146 ceph-mon[26954]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-03-24T11:45:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:20.142 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"fsid":"0ddfea52-e9d3-11ee-95cd-87774f69a715","created":"2024-03-24T11:39:07.523110+0000","modified":"2024-03-24T11:45:04.991995+0000","last_up_change":"2024-03-24T11:45:01.005358+0000","last_in_change":"2024-03-24T11:44:43.246825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:42:59.003957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9ac184de-5c35-4015-b99e-75966907f527","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6803","nonce":249424652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6805","nonce":249424652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6809","nonce":249424652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":249424652},{"type":"v1","addr":"172.21.15.32:6807","nonce":249424652}]},"public_addr":"172.21.15.32:6803/249424652","cluster_addr":"172.21.15.32:6805/249424652","heartbeat_back_addr":"172.21.15.32:6809/249424652","heartbeat_front_addr":"172.21.15.32:6807/249424652","state":["exists","up"]},{"osd":1,"uuid":"5d4ec7af-f4dd-4201-aae9-cb21009558dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6811","nonce":3261903682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6813","nonce":3261903682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6817","nonce":3261903682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3261903682},{"type":"v1","addr":"172.21.15.32:6815","nonce":3261903682}]},"public_addr":"172.21.15.32:6811/3261903682","cluster_addr":"172.21.15.32:6813/3261903682","heartbeat_back_addr":"172.21.15.32:6817/3261903682","heartbeat_front_addr":"172.21.15.32:6815/3261903682","state":["exists","up"]},{"osd":2,"uuid":"e4a7326a-5ac8-4e97-bff3-a4291969db49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6819","nonce":3671793277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6821","nonce":3671793277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6825","nonce":3671793277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3671793277},{"type":"v1","addr":"172.21.15.32:6823","nonce":3671793277}]},"public_addr":"172.21.15.32:6819/3671793277","cluster_addr":"172.21.15.32:6821/3671793277","heartbeat_back_addr":"172.21.15.32:6825/3671793277","heartbeat_front_addr":"172.21.15.32:6823/3671793277","state":["exists","up"]},{"osd":3,"uuid":"15e87c2b-4340-41e4-9334-398c0931f237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6827","nonce":153369643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6829","nonce":153369643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6833","nonce":153369643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":153369643},{"type":"v1","addr":"172.21.15.32:6831","nonce":153369643}]},"public_addr":"172.21.15.32:6827/153369643","cluster_addr":"172.21.15.32:6829/153369643","heartbeat_back_addr":"172.21.15.32:6833/153369643","heartbeat_front_addr":"172.21.15.32:6831/153369643","state":["exists","up"]},{"osd":4,"uuid":"7f8a3113-710d-4ac3-a14d-524ca4df7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6801","nonce":3324113540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6803","nonce":3324113540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6807","nonce":3324113540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3324113540},{"type":"v1","addr":"172.21.15.146:6805","nonce":3324113540}]},"public_addr":"172.21.15.146:6801/3324113540","cluster_addr":"172.21.15.146:6803/3324113540","heartbeat_back_addr":"172.21.15.146:6807/3324113540","heartbeat_front_addr":"172.21.15.146:6805/3324113540","state":["exists","up"]},{"osd":5,"uuid":"22947993-6c1a-44b3-a06f-ac0c0dc4f1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6809","nonce":1567683755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6811","nonce":1567683755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6815","nonce":1567683755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1567683755},{"type":"v1","addr":"172.21.15.146:6813","nonce":1567683755}]},"public_addr":"172.21.15.146:6809/1567683755","cluster_addr":"172.21.15.146:6811/1567683755","heartbeat_back_addr":"172.21.15.146:6815/1567683755","heartbeat_front_addr":"172.21.15.146:6813/1567683755","state":["exists","up"]},{"osd":6,"uuid":"badcd75d-dd8b-4c2a-bdfb-c46b056654bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6817","nonce":1086572132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6819","nonce":1086572132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6823","nonce":1086572132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1086572132},{"type":"v1","addr":"172.21.15.146:6821","nonce":1086572132}]},"public_addr":"172.21.15.146:6817/1086572132","cluster_addr":"172.21.15.146:6819/1086572132","heartbeat_back_addr":"172.21.15.146:6823/1086572132","heartbeat_front_addr":"172.21.15.146:6821/1086572132","state":["exists","up"]},{"osd":7,"uuid":"39223732-7eb6-403f-bf5a-3a74f6621ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6825","nonce":2888776911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6827","nonce":2888776911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6831","nonce":2888776911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2888776911},{"type":"v1","addr":"172.21.15.146:6829","nonce":2888776911}]},"public_addr":"172.21.15.146:6825/2888776911","cluster_addr":"172.21.15.146:6827/2888776911","heartbeat_back_addr":"172.21.15.146:6831/2888776911","heartbeat_front_addr":"172.21.15.146:6829/2888776911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:59.820459+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:31.517819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:57.327174+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:43:22.660002+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:43:44.238723+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:09.409360+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:34.224000+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:44:59.076527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2430898427":"2024-03-25T11:40:56.627105+0000","172.21.15.32:6801/810062755":"2024-03-25T11:40:56.627105+0000","172.21.15.32:6801/2255656955":"2024-03-25T11:40:05.741283+0000","172.21.15.32:6800/2255656955":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/3060386269":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/1248672124":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/153620913":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/3749458703":"2024-03-25T11:40:05.741283+0000","172.21.15.32:0/1364378637":"2024-03-25T11:39:36.628204+0000","172.21.15.32:0/3127983456":"2024-03-25T11:40:05.741283+0000","172.21.15.32:6800/1288508470":"2024-03-25T11:39:36.628204+0000","172.21.15.32:6800/810062755":"2024-03-25T11:40:56.627105+0000","172.21.15.32:0/2072521784":"2024-03-25T11:39:36.628204+0000","172.21.15.32:0/72936805":"2024-03-25T11:39:36.628204+0000","172.21.15.32:6801/1288508470":"2024-03-25T11:39:36.628204+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:45:20.444 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:20 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1754339440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:45:20.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:20 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1754339440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:45:20.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:45:20.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:45:20.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:45:20.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T11:45:20.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T11:45:20.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T11:45:20.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T11:45:20.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T11:45:21.161 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.164 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.164 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.168 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.169 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.172 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.173 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.174 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:21 smithi146 ceph-mon[26954]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:21.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:21 smithi032 ceph-mon[24793]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:23.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:22 smithi032 ceph-mon[24793]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:23.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:23 smithi146 ceph-mon[26954]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:24 smithi146 ceph-mon[26954]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:25.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:24 smithi032 ceph-mon[24793]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:26.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:26 smithi032 ceph-mon[24793]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:26 smithi146 ceph-mon[26954]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:28.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:28 smithi032 ceph-mon[24793]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:29.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:28 smithi146 ceph-mon[26954]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:29.540 INFO:teuthology.orchestra.run.smithi032.stdout:201863462925 2024-03-24T11:45:29.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T11:45:31.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:30 smithi032 ceph-mon[24793]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:31.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:30 smithi146 ceph-mon[26954]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:32.743 INFO:teuthology.orchestra.run.smithi032.stdout:38654705708 2024-03-24T11:45:32.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:45:33.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:33 smithi146 ceph-mon[26954]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:33.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:33 smithi032 ceph-mon[24793]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:33.432 INFO:teuthology.orchestra.run.smithi032.stdout:81604378658 2024-03-24T11:45:33.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:45:33.781 INFO:teuthology.orchestra.run.smithi032.stdout:133143986198 2024-03-24T11:45:33.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T11:45:34.162 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:34.262 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:34.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:34 smithi032 ceph-mon[24793]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:34 smithi146 ceph-mon[26954]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:35.719 INFO:teuthology.orchestra.run.smithi032.stdout:60129542183 2024-03-24T11:45:35.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:45:37.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:36 smithi032 ceph-mon[24793]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:36 smithi146 ceph-mon[26954]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:37.150 INFO:teuthology.orchestra.run.smithi032.stdout:111669149724 2024-03-24T11:45:37.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T11:45:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:38.654 INFO:teuthology.orchestra.run.smithi032.stdout:201863462927 2024-03-24T11:45:38.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:38 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/4166904227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:45:39.002 INFO:teuthology.orchestra.run.smithi032.stdout:236223201291 2024-03-24T11:45:39.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T11:45:39.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:38 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/4166904227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:45:39.236 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:39.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:39 smithi032 ceph-mon[24793]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:40.035 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:40.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:39 smithi146 ceph-mon[26954]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:41.036 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:40 smithi146 ceph-mon[26954]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:41.141 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:40 smithi032 ceph-mon[24793]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:41.678 INFO:teuthology.orchestra.run.smithi032.stdout:167503724564 2024-03-24T11:45:41.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T11:45:42.466 INFO:teuthology.orchestra.run.smithi032.stdout:38654705710 2024-03-24T11:45:43.081 INFO:teuthology.orchestra.run.smithi032.stdout:81604378660 2024-03-24T11:45:43.081 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:43 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3565562728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:45:43.081 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:43 smithi032 ceph-mon[24793]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:43.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705708 got 38654705710 for osd.0 2024-03-24T11:45:43.305 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:43 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3565562728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:45:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:43 smithi146 ceph-mon[26954]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:44.365 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:44 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3768222574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:45:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:44.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:44 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3768222574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:45:44.396 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462925 got 201863462927 for osd.6 2024-03-24T11:45:44.396 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:45.280 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:45 smithi032 ceph-mon[24793]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:45.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:45 smithi146 ceph-mon[26954]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:45.568 INFO:teuthology.orchestra.run.smithi032.stdout:133143986201 2024-03-24T11:45:45.738 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378658 got 81604378660 for osd.2 2024-03-24T11:45:45.738 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:46.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:46 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3858316380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:45:46.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:46 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3858316380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:45:46.519 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986198 got 133143986201 for osd.4 2024-03-24T11:45:46.519 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:47.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:47 smithi032 ceph-mon[24793]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:47.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:47 smithi146 ceph-mon[26954]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:48.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:48 smithi032 ceph-mon[24793]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:49.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:48 smithi146 ceph-mon[26954]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:49.935 INFO:teuthology.orchestra.run.smithi032.stdout:111669149728 2024-03-24T11:45:50.128 INFO:teuthology.orchestra.run.smithi032.stdout:60129542186 2024-03-24T11:45:50.173 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:49 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3376489249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:45:50.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:49 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3376489249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:45:50.683 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-03-24T11:45:50.683 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:51.105 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149724 got 111669149728 for osd.3 2024-03-24T11:45:51.105 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:51.288 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:50 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/327165540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:45:51.288 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:50 smithi032 ceph-mon[24793]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:51.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:50 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/327165540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:45:51.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:50 smithi146 ceph-mon[26954]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:51.498 INFO:teuthology.orchestra.run.smithi032.stdout:167503724567 2024-03-24T11:45:52.260 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:51 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1166884341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:45:52.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724564 got 167503724567 for osd.5 2024-03-24T11:45:52.286 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:52.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:51 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1166884341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:45:52.874 INFO:teuthology.orchestra.run.smithi032.stdout:236223201293 2024-03-24T11:45:53.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:53 smithi032 ceph-mon[24793]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:53.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:53 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2469075946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:45:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:53 smithi146 ceph-mon[26954]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:53 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/2469075946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:45:53.406 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201291 got 236223201293 for osd.7 2024-03-24T11:45:53.407 DEBUG:teuthology.parallel:result is None 2024-03-24T11:45:53.407 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:45:53.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:45:53.627 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:54.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:54 smithi032 ceph-mon[24793]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:55.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:54 smithi146 ceph-mon[26954]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:55.331 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-24T11:45:55.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:55 smithi032 ceph-mon[24793]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:45:55.886 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-03-24T11:45:54.732051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48868,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912348,"statfs":{"total":767960285184,"available":767910244352,"internally_reserved":0,"allocated":6459392,"data_stored":3547149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002132"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-24T11:45:04.998182+0000","last_change":"2024-03-24T11:45:04.998182+0000","last_active":"2024-03-24T11:45:04.998182+0000","last_peered":"2024-03-24T11:45:04.998182+0000","last_clean":"2024-03-24T11:45:04.998182+0000","last_became_active":"2024-03-24T11:45:04.997596+0000","last_became_peered":"2024-03-24T11:45:04.997596+0000","last_unstale":"2024-03-24T11:45:04.998182+0000","last_undegraded":"2024-03-24T11:45:04.998182+0000","last_fullsized":"2024-03-24T11:45:04.998182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:59.380227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:59.380227+0000","last_clean_scrub_stamp":"2024-03-24T11:42:59.380227+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:08:57.596392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.69999999999999996}]},{"osd":1,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.629},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.73299999999999998}]},{"osd":2,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.76600000000000001}]},{"osd":3,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.80100000000000005}]},{"osd":4,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.71699999999999997}]},{"osd":5,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.79200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.35499999999999998,"15min":0.33900000000000002},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.375,"15min":0.35299999999999998},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.314}]},{"osd":2,"last update":"Sun Mar 24 11:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.378,"15min":0.36599999999999999},"min":{"1min":0.32600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.36899999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.30599999999999999}]},{"osd":3,"last update":"Sun Mar 24 11:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.46300000000000002,"15min":0.46100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.372}]},{"osd":4,"last update":"Sun Mar 24 11:45:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.628},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77200000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.47099999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.47099999999999997}]},{"osd":5,"last update":"Sun Mar 24 11:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.501}]},{"osd":6,"last update":"Sun Mar 24 11:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.316},"min":{"1min":0.24099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.34899999999999998,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.53100000000000003,"5min":0.621,"15min":0.621},"last":0.30599999999999999}]},{"osd":2,"last update":"Sun Mar 24 11:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38,"15min":0.373},"min":{"1min":0.28299999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.40100000000000002},"min":{"1min":0.33700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.35599999999999998}]},{"osd":3,"last update":"Sun Mar 24 11:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39700000000000002}]},{"osd":4,"last update":"Sun Mar 24 11:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78900000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.498}]},{"osd":5,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.505}]},{"osd":6,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.378,"15min":0.378},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51000000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.36299999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40600000000000003}]},{"osd":1,"last update":"Sun Mar 24 11:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.373},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.28199999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Sun Mar 24 11:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.55800000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.498}]},{"osd":4,"last update":"Sun Mar 24 11:45:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":0.74199999999999999,"15min":0.63500000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":18.931000000000001,"5min":18.931000000000001,"15min":18.931000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.92800000000000005}]},{"osd":5,"last update":"Sun Mar 24 11:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.91700000000000004}]},{"osd":6,"last update":"Sun Mar 24 11:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.89900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Sun Mar 24 11:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57999999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57199999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.37}]},{"osd":2,"last update":"Sun Mar 24 11:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.31900000000000001}]},{"osd":4,"last update":"Sun Mar 24 11:44:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.66000000000000003}]},{"osd":5,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Sun Mar 24 11:45:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55300000000000005,"15min":0.54600000000000004},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.69299999999999995}]},{"osd":2,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42699999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.42699999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.63}]},{"osd":3,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.434,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.45100000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.72299999999999998}]},{"osd":5,"last update":"Sun Mar 24 11:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.433}]},{"osd":6,"last update":"Sun Mar 24 11:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.434},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53600000000000003}]},{"osd":3,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.496},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60199999999999998}]},{"osd":4,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.55300000000000005}]},{"osd":6,"last update":"Sun Mar 24 11:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:45:55.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:45:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:55 smithi146 ceph-mon[26954]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:45:57.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:56 smithi032 ceph-mon[24793]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:57.076 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:45:57.076 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:45:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:56 smithi146 ceph-mon[26954]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:45:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:45:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:45:57.809 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-24T11:45:58.386 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-03-24T11:45:56.732378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48868,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912348,"statfs":{"total":767960285184,"available":767910244352,"internally_reserved":0,"allocated":6459392,"data_stored":3547149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002005"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-24T11:45:04.998182+0000","last_change":"2024-03-24T11:45:04.998182+0000","last_active":"2024-03-24T11:45:04.998182+0000","last_peered":"2024-03-24T11:45:04.998182+0000","last_clean":"2024-03-24T11:45:04.998182+0000","last_became_active":"2024-03-24T11:45:04.997596+0000","last_became_peered":"2024-03-24T11:45:04.997596+0000","last_unstale":"2024-03-24T11:45:04.998182+0000","last_undegraded":"2024-03-24T11:45:04.998182+0000","last_fullsized":"2024-03-24T11:45:04.998182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:59.380227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:59.380227+0000","last_clean_scrub_stamp":"2024-03-24T11:42:59.380227+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:08:57.596392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.77000000000000002}]},{"osd":1,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.71699999999999997}]},{"osd":2,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.64800000000000002}]},{"osd":3,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.63400000000000001}]},{"osd":4,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.66200000000000003}]},{"osd":5,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.35499999999999998,"15min":0.33900000000000002},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.375,"15min":0.35299999999999998},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.314}]},{"osd":2,"last update":"Sun Mar 24 11:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.378,"15min":0.36599999999999999},"min":{"1min":0.32600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.36899999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.30599999999999999}]},{"osd":3,"last update":"Sun Mar 24 11:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.46300000000000002,"15min":0.46100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.372}]},{"osd":4,"last update":"Sun Mar 24 11:45:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.628},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77200000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.47099999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.47099999999999997}]},{"osd":5,"last update":"Sun Mar 24 11:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.501}]},{"osd":6,"last update":"Sun Mar 24 11:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.316},"min":{"1min":0.24099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.34899999999999998,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.53100000000000003,"5min":0.621,"15min":0.621},"last":0.30599999999999999}]},{"osd":2,"last update":"Sun Mar 24 11:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38,"15min":0.373},"min":{"1min":0.28299999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.40100000000000002},"min":{"1min":0.33700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.35599999999999998}]},{"osd":3,"last update":"Sun Mar 24 11:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39700000000000002}]},{"osd":4,"last update":"Sun Mar 24 11:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78900000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.498}]},{"osd":5,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.505}]},{"osd":6,"last update":"Sun Mar 24 11:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.378,"15min":0.378},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51000000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.36299999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40600000000000003}]},{"osd":1,"last update":"Sun Mar 24 11:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.373},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.28199999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Sun Mar 24 11:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.55800000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.498}]},{"osd":4,"last update":"Sun Mar 24 11:45:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":0.74199999999999999,"15min":0.63500000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":18.931000000000001,"5min":18.931000000000001,"15min":18.931000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.92800000000000005}]},{"osd":5,"last update":"Sun Mar 24 11:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.91700000000000004}]},{"osd":6,"last update":"Sun Mar 24 11:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.89900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33300000000000002}]},{"osd":1,"last update":"Sun Mar 24 11:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57999999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57199999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Mar 24 11:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.378}]},{"osd":4,"last update":"Sun Mar 24 11:45:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.38700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.45200000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76900000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.85899999999999999}]},{"osd":5,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.66600000000000004}]},{"osd":6,"last update":"Sun Mar 24 11:45:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.55700000000000005}]},{"osd":1,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55300000000000005,"15min":0.54600000000000004},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51600000000000001}]},{"osd":2,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42699999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.42699999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.58299999999999996}]},{"osd":3,"last update":"Sun Mar 24 11:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.434,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.496},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.45100000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.48599999999999999}]},{"osd":5,"last update":"Sun Mar 24 11:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.442}]},{"osd":6,"last update":"Sun Mar 24 11:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724569,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.748}]},{"osd":2,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.622},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.72099999999999997}]},{"osd":3,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.80800000000000005}]},{"osd":4,"last update":"Sun Mar 24 11:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.47499999999999998}]},{"osd":6,"last update":"Sun Mar 24 11:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.498},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:45:58.388 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:45:58.388 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:45:58.388 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:45:58.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:45:58.610 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/mon.smithi032/config 2024-03-24T11:45:58.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:58 smithi032 ceph-mon[24793]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:45:58.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:45:58 smithi032 ceph-mon[24793]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:58 smithi146 ceph-mon[26954]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:45:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:45:58 smithi146 ceph-mon[26954]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:00.386 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:46:00.386 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:46:00.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:00 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2743557752' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:46:00.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:00 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/2743557752' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:46:01.013 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:46:01.014 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:46:01.014 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:46:01.023 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:46:01.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T11:46:01.626 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:01 smithi032 ceph-mon[24793]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:01.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:01 smithi146 ceph-mon[26954]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:02.931 INFO:teuthology.orchestra.run.smithi032.stdout:Backend: cephadm 2024-03-24T11:46:02.931 INFO:teuthology.orchestra.run.smithi032.stdout:Available: Yes 2024-03-24T11:46:02.931 INFO:teuthology.orchestra.run.smithi032.stdout:Paused: No 2024-03-24T11:46:03.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:03 smithi032 ceph-mon[24793]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:03.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:03 smithi146 ceph-mon[26954]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:03.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T11:46:04.165 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:04 smithi032 ceph-mon[24793]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:04.165 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:04.165 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:04.165 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:04.165 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:04 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:04 smithi146 ceph-mon[26954]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:04 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:05.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:05 smithi032 ceph-mon[24793]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:05.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:05 smithi146 ceph-mon[26954]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:46:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 34e04a907cd2 2024-03-24T11:46:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:46:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (4m) 65s ago 4m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:46:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (4m) 2m ago 5m 47.8M - 8.3.5 dad864ee21e9 3669a5af1bd5 2024-03-24T11:46:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 7b699f0e61e8 2024-03-24T11:46:05.409 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (4m) 65s ago 4m 413M - 17.2.0 e1d6a67b021e 546cb874d84e 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 2m ago 6m 41.4M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (4m) 65s ago 4m 39.7M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (5m) 2m ago 5m 17.5M - 1dbe0e931976 8bec60c4b714 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (4m) 65s ago 4m 15.5M - 1dbe0e931976 9c0cab415cfb 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (4m) 2m ago 4m 35.4M 3217M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 2m ago 3m 34.0M 3217M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (3m) 2m ago 3m 33.7M 3217M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (2m) 2m ago 2m 13.4M 3217M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (2m) 65s ago 2m 32.9M 3984M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (2m) 65s ago 2m 33.4M 3984M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (95s) 65s ago 95s 31.6M 3984M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (70s) 65s ago 70s 13.4M 3984M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:46:05.410 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (4m) 2m ago 4m 51.6M - 514e6a882f6e 5c09869266dc 2024-03-24T11:46:06.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T11:46:06.327 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:06 smithi032 ceph-mon[24793]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:06.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:06 smithi146 ceph-mon[26954]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:07.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:07 smithi032 ceph-mon[24793]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:07 smithi146 ceph-mon[26954]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:07.903 INFO:teuthology.orchestra.run.smithi032.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T11:46:07.903 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-24T11:46:07.904 INFO:teuthology.orchestra.run.smithi032.stdout:crash 2/2 2m ago 6m * 2024-03-24T11:46:07.904 INFO:teuthology.orchestra.run.smithi032.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-24T11:46:07.904 INFO:teuthology.orchestra.run.smithi032.stdout:mgr 2/2 2m ago 6m count:2 2024-03-24T11:46:07.904 INFO:teuthology.orchestra.run.smithi032.stdout:mon 2/2 2m ago 5m smithi032:172.21.15.32=smithi032;smithi146:172.21.15.146=smithi146;count:2 2024-03-24T11:46:07.904 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-24T11:46:07.904 INFO:teuthology.orchestra.run.smithi032.stdout:osd 8 2m ago - 2024-03-24T11:46:07.904 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-24T11:46:08.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T11:46:09.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:08 smithi032 ceph-mon[24793]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:09.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:08 smithi032 ceph-mon[24793]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:09.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:08 smithi146 ceph-mon[26954]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:09.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:08 smithi146 ceph-mon[26954]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:10.370 INFO:teuthology.orchestra.run.smithi032.stdout:HOST ADDR LABELS STATUS 2024-03-24T11:46:10.370 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 172.21.15.32 2024-03-24T11:46:10.370 INFO:teuthology.orchestra.run.smithi032.stdout:smithi146 172.21.15.146 2024-03-24T11:46:10.370 INFO:teuthology.orchestra.run.smithi032.stdout:2 hosts in cluster 2024-03-24T11:46:10.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:10 smithi032 ceph-mon[24793]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:11.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T11:46:11.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:10 smithi146 ceph-mon[26954]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:11 smithi032 ceph-mon[24793]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:12.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:11 smithi146 ceph-mon[26954]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:12.929 INFO:teuthology.orchestra.run.smithi032.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-03-24T11:46:12.929 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402DN480DGN 480G LVM detected, locked 2024-03-24T11:46:12.930 INFO:teuthology.orchestra.run.smithi032.stdout:smithi146 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GB400BGN 400G LVM detected, locked 2024-03-24T11:46:13.506 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-24T11:46:13.516 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:46:13.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-24T11:46:13.545 INFO:teuthology.orchestra.run.smithi032.stderr:+ systemctl stop nfs-server 2024-03-24T11:46:13.553 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi146.front.sepia.ceph.com 2024-03-24T11:46:13.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-24T11:46:13.585 INFO:teuthology.orchestra.run.smithi146.stderr:+ systemctl stop nfs-server 2024-03-24T11:46:13.594 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:46:13.602 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:46:13.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph fs volume create foofs' 2024-03-24T11:46:14.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:14 smithi032 ceph-mon[24793]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:14.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:14 smithi032 ceph-mon[24793]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:14 smithi146 ceph-mon[26954]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:14 smithi146 ceph-mon[26954]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:16.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:16 smithi146 ceph-mon[26954]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:16.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:16 smithi146 ceph-mon[26954]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:16.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:16 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-24T11:46:16.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:16 smithi032 ceph-mon[24793]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:16.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:16 smithi032 ceph-mon[24793]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:16.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:16 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-24T11:46:17.349 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:17 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-24T11:46:17.349 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:17 smithi032 ceph-mon[24793]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:46:17.349 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:17 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-24T11:46:17.349 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:17 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[24789]: 2024-03-24T11:46:17.023+0000 7f71f360b700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:46:17.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:17 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-24T11:46:17.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:17 smithi146 ceph-mon[26954]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:46:17.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:17 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-24T11:46:17.655 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-03-24T11:46:17.665 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-03-24T11:46:17.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T11:46:18.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: pgmap v208: 33 pgs: 10 creating+peering, 1 active+clean, 22 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: fsmap foofs:0 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: Saving service mds.foofs spec with placement count:2 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:18.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:18.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:18.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:46:18.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:46:18.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:18.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:18 smithi032 ceph-mon[24793]: Deploying daemon mds.foofs.smithi032.qzdcjy on smithi032 2024-03-24T11:46:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: pgmap v208: 33 pgs: 10 creating+peering, 1 active+clean, 22 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-24T11:46:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:46:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-24T11:46:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:46:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: fsmap foofs:0 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: Saving service mds.foofs spec with placement count:2 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:46:18.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:18.387 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:18 smithi146 ceph-mon[26954]: Deploying daemon mds.foofs.smithi032.qzdcjy on smithi032 2024-03-24T11:46:19.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:19 smithi032 ceph-mon[24793]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:46:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:19 smithi146 ceph-mon[26954]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:46:20.285 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:20 smithi032 ceph-mon[24793]: pgmap v212: 65 pgs: 3 active+clean, 22 creating+peering, 40 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:20.286 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:20 smithi032 ceph-mon[24793]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:46:20.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:20 smithi146 ceph-mon[26954]: pgmap v212: 65 pgs: 3 active+clean, 22 creating+peering, 40 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:20.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:20 smithi146 ceph-mon[26954]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:46:20.674 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:46:20.674 INFO:teuthology.orchestra.run.smithi032.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T11:39:55.322739Z", "last_refresh": "2024-03-24T11:43:25.136190Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T11:39:49.097615Z", "last_refresh": "2024-03-24T11:43:25.136261Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T11:39:52.284325Z", "last_refresh": "2024-03-24T11:43:25.136327Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T11:46:17.046628Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T11:46:17.041029Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T11:39:47.601685Z", "last_refresh": "2024-03-24T11:43:25.136099Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi032:172.21.15.32=smithi032", "smithi146:172.21.15.146=smithi146"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T11:40:37.521788Z", "last_refresh": "2024-03-24T11:43:25.135956Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T11:39:53.434504Z", "last_refresh": "2024-03-24T11:43:25.136391Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T11:43:25.136520Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T11:39:50.497551Z", "last_refresh": "2024-03-24T11:43:25.136455Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T11:46:21.289 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:21 smithi032 ceph-mon[24793]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:21.289 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:21 smithi032 ceph-mon[24793]: pgmap v214: 65 pgs: 25 active+clean, 22 creating+peering, 18 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:21 smithi146 ceph-mon[26954]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:21 smithi146 ceph-mon[26954]: pgmap v214: 65 pgs: 25 active+clean, 22 creating+peering, 18 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:21.412 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-24T11:46:22.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: mds.? [v2:172.21.15.32:6834/3101395579,v1:172.21.15.32:6835/3101395579] up:boot 2024-03-24T11:46:22.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: daemon mds.foofs.smithi032.qzdcjy assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-24T11:46:22.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:46:22.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: Cluster is now healthy 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: fsmap foofs:0 1 up:standby 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:creating} 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: daemon mds.foofs.smithi032.qzdcjy is now active in filesystem foofs as rank 0 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:22.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:22 smithi146 ceph-mon[26954]: Deploying daemon mds.foofs.smithi146.xsrsue on smithi146 2024-03-24T11:46:22.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: mds.? [v2:172.21.15.32:6834/3101395579,v1:172.21.15.32:6835/3101395579] up:boot 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: daemon mds.foofs.smithi032.qzdcjy assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: Cluster is now healthy 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: fsmap foofs:0 1 up:standby 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:creating} 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: daemon mds.foofs.smithi032.qzdcjy is now active in filesystem foofs as rank 0 2024-03-24T11:46:22.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:22.444 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:46:22.444 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:46:22.444 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:22.444 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:22 smithi032 ceph-mon[24793]: Deploying daemon mds.foofs.smithi146.xsrsue on smithi146 2024-03-24T11:46:23.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:23 smithi146 ceph-mon[26954]: mds.? [v2:172.21.15.32:6834/3101395579,v1:172.21.15.32:6835/3101395579] up:active 2024-03-24T11:46:23.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:23 smithi146 ceph-mon[26954]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 2024-03-24T11:46:23.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:23 smithi146 ceph-mon[26954]: pgmap v215: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-03-24T11:46:23.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:23 smithi032 ceph-mon[24793]: mds.? [v2:172.21.15.32:6834/3101395579,v1:172.21.15.32:6835/3101395579] up:active 2024-03-24T11:46:23.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:23 smithi032 ceph-mon[24793]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 2024-03-24T11:46:23.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:23 smithi032 ceph-mon[24793]: pgmap v215: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-03-24T11:46:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:46:24.411 INFO:teuthology.orchestra.run.smithi032.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T11:39:55.322739Z", "last_refresh": "2024-03-24T11:43:25.136190Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T11:39:49.097615Z", "last_refresh": "2024-03-24T11:43:25.136261Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T11:39:52.284325Z", "last_refresh": "2024-03-24T11:43:25.136327Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T11:46:17.046628Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T11:46:17.041029Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T11:39:47.601685Z", "last_refresh": "2024-03-24T11:43:25.136099Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi032:172.21.15.32=smithi032", "smithi146:172.21.15.146=smithi146"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T11:40:37.521788Z", "last_refresh": "2024-03-24T11:43:25.135956Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T11:39:53.434504Z", "last_refresh": "2024-03-24T11:43:25.136391Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T11:43:25.136520Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T11:39:50.497551Z", "last_refresh": "2024-03-24T11:43:25.136455Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T11:46:24.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:24.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:24.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:24.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:24.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:boot 2024-03-24T11:46:24.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:46:24.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:46:24.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:24 smithi032 ceph-mon[24793]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:boot 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:46:24.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:24 smithi146 ceph-mon[26954]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:46:25.672 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-24T11:46:25.908 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:25 smithi032 ceph-mon[24793]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:25.908 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:25 smithi032 ceph-mon[24793]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-24T11:46:25.964 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:25 smithi146 ceph-mon[26954]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:25.965 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:25 smithi146 ceph-mon[26954]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-24T11:46:26.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T11:46:26.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:26 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:26.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:26 smithi032 ceph-mon[24793]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-24T11:46:26.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:26 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:26 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:26 smithi146 ceph-mon[26954]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-24T11:46:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:26 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:28.716 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:46:28.717 INFO:teuthology.orchestra.run.smithi032.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T11:39:55.322739Z", "last_refresh": "2024-03-24T11:43:25.136190Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T11:39:49.097615Z", "last_refresh": "2024-03-24T11:43:25.136261Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T11:39:52.284325Z", "last_refresh": "2024-03-24T11:43:25.136327Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T11:46:17.046628Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T11:46:17.041029Z", "last_refresh": "2024-03-24T11:46:25.887807Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T11:39:47.601685Z", "last_refresh": "2024-03-24T11:43:25.136099Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi032:172.21.15.32=smithi032", "smithi146:172.21.15.146=smithi146"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T11:40:37.521788Z", "last_refresh": "2024-03-24T11:43:25.135956Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T11:39:53.434504Z", "last_refresh": "2024-03-24T11:43:25.136391Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T11:43:25.136520Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T11:39:50.497551Z", "last_refresh": "2024-03-24T11:43:25.136455Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T11:46:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:28 smithi146 ceph-mon[26954]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:28 smithi146 ceph-mon[26954]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T11:46:29.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:28 smithi032 ceph-mon[24793]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:29.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:28 smithi032 ceph-mon[24793]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T11:46:29.498 INFO:tasks.cephadm:mds.foofs has 1/2 2024-03-24T11:46:30.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T11:46:31.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:31.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:31.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:31.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:31.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:31.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:31 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:31.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:31 smithi032 ceph-mon[24793]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T11:46:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:31 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:31 smithi146 ceph-mon[26954]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T11:46:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:46:32.463 INFO:teuthology.orchestra.run.smithi032.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T11:39:55.322739Z", "last_refresh": "2024-03-24T11:46:30.674466Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T11:39:49.097615Z", "last_refresh": "2024-03-24T11:46:25.887183Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T11:39:52.284325Z", "last_refresh": "2024-03-24T11:46:30.674600Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T11:46:17.046628Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T11:46:17.041029Z", "last_refresh": "2024-03-24T11:46:25.887807Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T11:39:47.601685Z", "last_refresh": "2024-03-24T11:46:25.887329Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi032:172.21.15.32=smithi032", "smithi146:172.21.15.146=smithi146"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T11:40:37.521788Z", "last_refresh": "2024-03-24T11:46:25.887404Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T11:39:53.434504Z", "last_refresh": "2024-03-24T11:46:25.887470Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T11:46:25.887535Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T11:39:50.497551Z", "last_refresh": "2024-03-24T11:46:30.674726Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T11:46:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:32 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:32 smithi146 ceph-mon[26954]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:32 smithi146 ceph-mon[26954]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T11:46:33.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:32 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:33.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:32 smithi032 ceph-mon[24793]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:46:33.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:32 smithi032 ceph-mon[24793]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T11:46:33.167 INFO:tasks.cephadm:mds.foofs has 2/2 2024-03-24T11:46:33.167 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:46:33.177 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:46:33.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-03-24T11:46:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:34 smithi146 ceph-mon[26954]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T11:46:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:34 smithi032 ceph-mon[24793]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T11:46:36.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:35 smithi146 ceph-mon[26954]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:36.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:35 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-24T11:46:36.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:35 smithi032 ceph-mon[24793]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:36.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:35 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-24T11:46:36.848 INFO:teuthology.orchestra.run.smithi032.stdout:NFS Cluster Created Successfully 2024-03-24T11:46:37.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:36 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-24T11:46:37.076 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:36 smithi032 ceph-mon[24793]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:46:37.076 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:36 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-24T11:46:37.076 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:36 smithi032 ceph-mon[24793]: pgmap v223: 97 pgs: 14 creating+peering, 18 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-03-24T11:46:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:36 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-24T11:46:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:36 smithi146 ceph-mon[26954]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:46:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:36 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-24T11:46:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:36 smithi146 ceph-mon[26954]: pgmap v223: 97 pgs: 14 creating+peering, 18 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-03-24T11:46:37.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-03-24T11:46:37.949 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: Saving service nfs.foo spec with placement count:2 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T11:46:37.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: Rados config object exists: conf-nfs.foo 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs-rgw 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:37 smithi032 ceph-mon[24793]: Deploying daemon nfs.foo.0.0.smithi032.lizmvs on smithi032 2024-03-24T11:46:38.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-24T11:46:38.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: Saving service nfs.foo spec with placement count:2 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T11:46:38.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T11:46:38.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: Rados config object exists: conf-nfs.foo 2024-03-24T11:46:38.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs-rgw 2024-03-24T11:46:38.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:46:38.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:46:38.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:38.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:37 smithi146 ceph-mon[26954]: Deploying daemon nfs.foo.0.0.smithi032.lizmvs on smithi032 2024-03-24T11:46:39.131 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:38 smithi032 ceph-mon[24793]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:46:39.131 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:38 smithi032 ceph-mon[24793]: pgmap v226: 97 pgs: 14 creating+peering, 13 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:39.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:38 smithi146 ceph-mon[26954]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:46:39.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:38 smithi146 ceph-mon[26954]: pgmap v226: 97 pgs: 14 creating+peering, 13 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:Invalid command: Unexpected argument '--clusterid' 2024-03-24T11:46:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-03-24T11:46:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:Error EINVAL: invalid command 2024-03-24T11:46:40.037 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:39 smithi032 ceph-mon[24793]: mgrmap e20: smithi032.xjdgbw(active, since 5m), standbys: smithi146.tfohfl 2024-03-24T11:46:40.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:39 smithi146 ceph-mon[26954]: mgrmap e20: smithi032.xjdgbw(active, since 5m), standbys: smithi146.tfohfl 2024-03-24T11:46:40.391 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:46:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "bind": "/fake", 2024-03-24T11:46:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "fs": "foofs", 2024-03-24T11:46:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/", 2024-03-24T11:46:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster": "foo", 2024-03-24T11:46:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "RW" 2024-03-24T11:46:40.392 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:46:40.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:40 smithi032 ceph-mon[24793]: from='client.14528 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:40.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:40 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-24T11:46:40.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:40 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-24T11:46:40.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:40 smithi032 ceph-mon[24793]: pgmap v227: 97 pgs: 14 creating+peering, 83 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 511 B/s wr, 0 op/s 2024-03-24T11:46:41.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-03-24T11:46:41.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:40 smithi146 ceph-mon[26954]: from='client.14528 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:41.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:40 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-24T11:46:41.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:40 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-24T11:46:41.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:40 smithi146 ceph-mon[26954]: pgmap v227: 97 pgs: 14 creating+peering, 83 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 511 B/s wr, 0 op/s 2024-03-24T11:46:42.644 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:42.644 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds 2024-03-24T11:46:42.644 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T11:46:42.645 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T11:46:42.645 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T11:46:42.645 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T11:46:42.645 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T11:46:42.645 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:42 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds 2024-03-24T11:46:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T11:46:42.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T11:46:42.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T11:46:42.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T11:46:42.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T11:46:42.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:42 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:43 smithi146 ceph-mon[26954]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 147 B/s rd, 590 B/s wr, 1 op/s 2024-03-24T11:46:43.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:43 smithi032 ceph-mon[24793]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 147 B/s rd, 590 B/s wr, 1 op/s 2024-03-24T11:46:44.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:44 smithi032 ceph-mon[24793]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T11:46:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T11:46:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:46:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:46:44.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:44 smithi146 ceph-mon[26954]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T11:46:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T11:46:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:46:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:46:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:45 smithi146 ceph-mon[26954]: Rados config object exists: conf-nfs.foo 2024-03-24T11:46:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:45 smithi146 ceph-mon[26954]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds-rgw 2024-03-24T11:46:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:45 smithi146 ceph-mon[26954]: Deploying daemon nfs.foo.1.0.smithi146.rjcwds on smithi146 2024-03-24T11:46:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:45 smithi146 ceph-mon[26954]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:45 smithi146 ceph-mon[26954]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 511 B/s wr, 0 op/s 2024-03-24T11:46:45.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:45 smithi032 ceph-mon[24793]: Rados config object exists: conf-nfs.foo 2024-03-24T11:46:45.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:45 smithi032 ceph-mon[24793]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds-rgw 2024-03-24T11:46:45.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:45 smithi032 ceph-mon[24793]: Deploying daemon nfs.foo.1.0.smithi146.rjcwds on smithi146 2024-03-24T11:46:45.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:45 smithi032 ceph-mon[24793]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:45.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:45 smithi032 ceph-mon[24793]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 511 B/s wr, 0 op/s 2024-03-24T11:46:46.866 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:46 smithi146 ceph-mon[26954]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:46.867 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:46 smithi146 ceph-mon[26954]: pgmap v230: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:46:47.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:46 smithi032 ceph-mon[24793]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:47.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:46 smithi032 ceph-mon[24793]: pgmap v230: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:46:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:47 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:47 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:48.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:47 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:48.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:47 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:48.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:47 smithi146 ceph-mon[26954]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:48.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:47 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:48.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:47 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:48.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:47 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:48.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:47 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:48.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:47 smithi032 ceph-mon[24793]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:49.291 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:48 smithi146 ceph-mon[26954]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:49.291 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:48 smithi146 ceph-mon[26954]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-24T11:46:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:48 smithi032 ceph-mon[24793]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:48 smithi032 ceph-mon[24793]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-24T11:46:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:50 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:50 smithi146 ceph-mon[26954]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:50 smithi146 ceph-mon[26954]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-24T11:46:51.144 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:50 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:51.145 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:50 smithi032 ceph-mon[24793]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:51.145 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:50 smithi032 ceph-mon[24793]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-24T11:46:52.555 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo ?:2049 2/2 2s ago 15s count:2 2024-03-24T11:46:52.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:52.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:52.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:52.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:52.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:52 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:52.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:52 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:46:53.396 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-24T11:46:53.406 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:46:53.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-03-24T11:46:53.435 INFO:teuthology.orchestra.run.smithi032.stderr:+ mkdir /mnt/foo 2024-03-24T11:46:53.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-03-24T11:46:53.520 INFO:teuthology.orchestra.run.smithi032.stderr:++ hostname 2024-03-24T11:46:53.520 INFO:teuthology.orchestra.run.smithi032.stderr:+ mount -t nfs smithi032:/fake /mnt/foo -o sync 2024-03-24T11:46:53.647 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:53 smithi032 ceph-mon[24793]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:53.647 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:53 smithi032 ceph-mon[24793]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-24T11:46:53.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:53 smithi146 ceph-mon[26954]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:53.889 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:53 smithi146 ceph-mon[26954]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-24T11:46:53.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-03-24T11:46:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:+ echo test 2024-03-24T11:46:54.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-03-24T11:46:54.089 INFO:teuthology.orchestra.run.smithi032.stderr:+ sync 2024-03-24T11:46:54.245 INFO:teuthology.run_tasks:Running task parallel... 2024-03-24T11:46:54.256 INFO:teuthology.task.parallel:starting parallel... 2024-03-24T11:46:54.257 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T11:46:54.257 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-24T11:46:54.257 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:46:54.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-24T11:46:54.258 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T11:46:54.258 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-24T11:46:54.259 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T11:46:54.259 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:46:54.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-03-24T11:46:54.326 INFO:teuthology.orchestra.run.smithi032.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-24T11:46:54.326 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:46:54.326 INFO:teuthology.orchestra.run.smithi032.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-24T11:46:54.327 INFO:teuthology.orchestra.run.smithi032.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-24T11:46:54.427 INFO:teuthology.orchestra.run.smithi032.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-24T11:46:54.428 INFO:teuthology.orchestra.run.smithi032.stdout:releasing clients 2024-03-24T11:46:54.942 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:54 smithi032 ceph-mon[24793]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-24T11:46:55.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:54 smithi146 ceph-mon[26954]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-24T11:46:55.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 292 53.91 MB/sec warmup 1 sec latency 44.147 ms 2024-03-24T11:46:56.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 592 50.99 MB/sec warmup 2 sec latency 14.517 ms 2024-03-24T11:46:56.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-03-24T11:46:56.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-24T11:46:56.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-03-24T11:46:56.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-24T11:46:56.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:56 smithi032 ceph-mon[24793]: pgmap v235: 97 pgs: 97 active+clean; 46 MiB data, 112 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-03-24T11:46:56.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:46:56.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:46:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-03-24T11:46:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-24T11:46:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-03-24T11:46:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-24T11:46:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:56 smithi146 ceph-mon[26954]: pgmap v235: 97 pgs: 97 active+clean; 46 MiB data, 112 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-03-24T11:46:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:46:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:46:57.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 1171 38.36 MB/sec warmup 3 sec latency 33.582 ms 2024-03-24T11:46:57.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-03-24T11:46:57.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-24T11:46:57.975 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-03-24T11:46:57.975 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:57 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-24T11:46:57.975 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:57 smithi032 ceph-mon[24793]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:46:57.975 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:57 smithi032 ceph-mon[24793]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:46:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-03-24T11:46:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-24T11:46:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-03-24T11:46:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:57 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-24T11:46:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:57 smithi146 ceph-mon[26954]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:46:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:57 smithi146 ceph-mon[26954]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:46:58.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-24T11:46:58.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 2360 34.86 MB/sec warmup 4 sec latency 13.114 ms 2024-03-24T11:46:58.978 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:46:58 smithi032 ceph-mon[24793]: pgmap v238: 97 pgs: 97 active+clean; 86 MiB data, 126 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 11 MiB/s wr, 409 op/s 2024-03-24T11:46:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:46:58 smithi146 ceph-mon[26954]: pgmap v238: 97 pgs: 97 active+clean; 86 MiB data, 126 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 11 MiB/s wr, 409 op/s 2024-03-24T11:46:59.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 3363 35.99 MB/sec warmup 5 sec latency 14.217 ms 2024-03-24T11:47:00.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 3934 33.34 MB/sec warmup 6 sec latency 97.178 ms 2024-03-24T11:47:01.031 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:00 smithi032 ceph-mon[24793]: pgmap v239: 97 pgs: 97 active+clean; 110 MiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 677 op/s 2024-03-24T11:47:01.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:00 smithi146 ceph-mon[26954]: pgmap v239: 97 pgs: 97 active+clean; 110 MiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 677 op/s 2024-03-24T11:47:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 4773 30.52 MB/sec warmup 7 sec latency 17.182 ms 2024-03-24T11:47:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 5757 28.81 MB/sec warmup 8 sec latency 24.396 ms 2024-03-24T11:47:03.040 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:03 smithi032 ceph-mon[24793]: pgmap v240: 97 pgs: 97 active+clean; 127 MiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 19 MiB/s wr, 1.00k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:03.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:03 smithi146 ceph-mon[26954]: pgmap v240: 97 pgs: 97 active+clean; 127 MiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 19 MiB/s wr, 1.00k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:03.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 6659 29.56 MB/sec warmup 9 sec latency 24.184 ms 2024-03-24T11:47:03.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-24T11:47:04.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 7428 29.52 MB/sec warmup 10 sec latency 16.553 ms 2024-03-24T11:47:05.048 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:04 smithi032 ceph-mon[24793]: pgmap v241: 97 pgs: 97 active+clean; 135 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 MiB/s wr, 908 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:04 smithi146 ceph-mon[26954]: pgmap v241: 97 pgs: 97 active+clean; 135 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 MiB/s wr, 908 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: 5 8250 27.91 MB/sec warmup 11 sec latency 40.725 ms 2024-03-24T11:47:06.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 9271 27.22 MB/sec warmup 12 sec latency 16.510 ms 2024-03-24T11:47:07.053 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:06 smithi032 ceph-mon[24793]: pgmap v242: 97 pgs: 97 active+clean; 142 MiB data, 333 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 1.02k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:06 smithi146 ceph-mon[26954]: pgmap v242: 97 pgs: 97 active+clean; 142 MiB data, 333 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 1.02k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 10216 27.74 MB/sec warmup 13 sec latency 15.972 ms 2024-03-24T11:47:08.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 10790 27.66 MB/sec warmup 14 sec latency 33.737 ms 2024-03-24T11:47:08.812 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:08 smithi032 ceph-mon[24793]: pgmap v243: 97 pgs: 97 active+clean; 151 MiB data, 386 MiB used, 715 GiB / 715 GiB avail; 7.8 KiB/s rd, 15 MiB/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:08.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph mgr module enable nfs --force' 2024-03-24T11:47:09.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:08 smithi146 ceph-mon[26954]: pgmap v243: 97 pgs: 97 active+clean; 151 MiB data, 386 MiB used, 715 GiB / 715 GiB avail; 7.8 KiB/s rd, 15 MiB/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:09.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 11198 26.13 MB/sec warmup 15 sec latency 25.796 ms 2024-03-24T11:47:10.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 11659 24.92 MB/sec warmup 16 sec latency 54.250 ms 2024-03-24T11:47:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:10 smithi032 ceph-mon[24793]: pgmap v244: 97 pgs: 97 active+clean; 151 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 1.09k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:10 smithi146 ceph-mon[26954]: pgmap v244: 97 pgs: 97 active+clean; 151 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 1.09k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:11.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 12290 24.41 MB/sec warmup 17 sec latency 27.021 ms 2024-03-24T11:47:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 12970 23.61 MB/sec warmup 18 sec latency 21.154 ms 2024-03-24T11:47:13.064 INFO:teuthology.orchestra.run.smithi032.stderr:module 'nfs' is already enabled 2024-03-24T11:47:13.129 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:13 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1613744012' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-24T11:47:13.129 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:13 smithi032 ceph-mon[24793]: pgmap v245: 97 pgs: 97 active+clean; 152 MiB data, 433 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.08k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:13 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1613744012' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-24T11:47:13.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:13 smithi146 ceph-mon[26954]: pgmap v245: 97 pgs: 97 active+clean; 152 MiB data, 433 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.08k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:47:13.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 13501 23.46 MB/sec warmup 19 sec latency 28.013 ms 2024-03-24T11:47:13.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-24T11:47:14.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:14 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1613744012' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-24T11:47:14.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:14 smithi032 ceph-mon[24793]: mgrmap e21: smithi032.xjdgbw(active, since 6m), standbys: smithi146.tfohfl 2024-03-24T11:47:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:14 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1613744012' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-24T11:47:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:14 smithi146 ceph-mon[26954]: mgrmap e21: smithi032.xjdgbw(active, since 6m), standbys: smithi146.tfohfl 2024-03-24T11:47:14.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 14143 23.62 MB/sec warmup 20 sec latency 39.531 ms 2024-03-24T11:47:15.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:15 smithi146 ceph-mon[26954]: pgmap v246: 97 pgs: 97 active+clean; 152 MiB data, 447 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 973 op/s 2024-03-24T11:47:15.152 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:15 smithi032 ceph-mon[24793]: pgmap v246: 97 pgs: 97 active+clean; 152 MiB data, 447 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 973 op/s 2024-03-24T11:47:15.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 14459 23.05 MB/sec warmup 21 sec latency 60.734 ms 2024-03-24T11:47:16.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 14950 22.24 MB/sec warmup 22 sec latency 23.103 ms 2024-03-24T11:47:16.915 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:16 smithi032 ceph-mon[24793]: pgmap v247: 97 pgs: 97 active+clean; 157 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 955 op/s 2024-03-24T11:47:17.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:16 smithi146 ceph-mon[26954]: pgmap v247: 97 pgs: 97 active+clean; 157 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 955 op/s 2024-03-24T11:47:17.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 15588 22.03 MB/sec warmup 23 sec latency 21.586 ms 2024-03-24T11:47:17.659 INFO:teuthology.orchestra.run.smithi032.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 16461 21.65 MB/sec warmup 24 sec latency 24.714 ms 2024-03-24T11:47:18.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:18.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:18.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:18.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:18.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:18.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:18.678 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:18.678 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:18 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:18.975 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-24T11:47:18.975 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T11:47:18.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-24T11:47:19.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:19.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:19.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:19.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:19.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:19.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:19.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:19.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:18 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:19.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 17025 21.65 MB/sec warmup 25 sec latency 31.081 ms 2024-03-24T11:47:19.698 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:19 smithi032 ceph-mon[24793]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:19.698 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:19 smithi032 ceph-mon[24793]: pgmap v248: 97 pgs: 97 active+clean; 168 MiB data, 495 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 907 op/s 2024-03-24T11:47:20.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:19 smithi146 ceph-mon[26954]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:20.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:19 smithi146 ceph-mon[26954]: pgmap v248: 97 pgs: 97 active+clean; 168 MiB data, 495 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 907 op/s 2024-03-24T11:47:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: 5 17619 21.79 MB/sec warmup 26 sec latency 30.634 ms 2024-03-24T11:47:20.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:20 smithi032 ceph-mon[24793]: pgmap v249: 97 pgs: 97 active+clean; 168 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 7.7 MiB/s wr, 671 op/s 2024-03-24T11:47:21.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:20 smithi146 ceph-mon[26954]: pgmap v249: 97 pgs: 97 active+clean; 168 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 7.7 MiB/s wr, 671 op/s 2024-03-24T11:47:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 18008 21.50 MB/sec warmup 27 sec latency 37.192 ms 2024-03-24T11:47:22.429 INFO:teuthology.orchestra.run.smithi032.stdout: 5 18378 20.85 MB/sec warmup 28 sec latency 34.598 ms 2024-03-24T11:47:23.085 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:47:23.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:23 smithi032 ceph-mon[24793]: pgmap v250: 97 pgs: 97 active+clean; 176 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 883 op/s 2024-03-24T11:47:23.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:23 smithi146 ceph-mon[26954]: pgmap v250: 97 pgs: 97 active+clean; 176 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 883 op/s 2024-03-24T11:47:23.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 18756 20.34 MB/sec warmup 29 sec latency 32.721 ms 2024-03-24T11:47:24.213 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 32s ago 6m 22.8M - ba2b418f427c 34e04a907cd2 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (6m) 32s ago 6m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (6m) 34s ago 6m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 32s ago 6m 53.4M - 8.3.5 dad864ee21e9 3669a5af1bd5 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (63s) 32s ago 63s 15.6M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (61s) 34s ago 60s 14.2M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:9283 running (8m) 32s ago 8m 461M - 17.2.0 e1d6a67b021e 7b699f0e61e8 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (6m) 34s ago 6m 414M - 17.2.0 e1d6a67b021e 546cb874d84e 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 32s ago 8m 46.5M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (6m) 34s ago 6m 37.9M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:47:24.214 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (43s) 32s ago 42s 46.9M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (37s) 34s ago 37s 52.9M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (6m) 32s ago 6m 19.6M - 1dbe0e931976 8bec60c4b714 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (6m) 34s ago 6m 16.0M - 1dbe0e931976 9c0cab415cfb 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (5m) 32s ago 5m 46.1M 3217M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (4m) 32s ago 4m 42.8M 3217M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 32s ago 4m 42.7M 3217M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (4m) 32s ago 4m 43.4M 3217M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (3m) 34s ago 3m 39.2M 3984M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (3m) 34s ago 3m 39.3M 3984M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (2m) 34s ago 2m 41.4M 3984M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (2m) 34s ago 2m 43.3M 3984M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:47:24.215 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 32s ago 5m 62.4M - 514e6a882f6e 5c09869266dc 2024-03-24T11:47:24.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:24 smithi032 ceph-mon[24793]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:24.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:24 smithi032 ceph-mon[24793]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:24 smithi146 ceph-mon[26954]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:24 smithi146 ceph-mon[26954]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 19214 20.09 MB/sec warmup 30 sec latency 36.754 ms 2024-03-24T11:47:24.879 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:47:24.879 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:47:24.879 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:47:24.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:24.879 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:47:24.879 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:47:24.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:47:24.880 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:47:25.281 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:25 smithi032 ceph-mon[24793]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:25.281 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:25 smithi032 ceph-mon[24793]: pgmap v251: 97 pgs: 97 active+clean; 178 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 814 op/s 2024-03-24T11:47:25.281 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:25 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/694227345' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:25.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:25 smithi146 ceph-mon[26954]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:25 smithi146 ceph-mon[26954]: pgmap v251: 97 pgs: 97 active+clean; 178 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 814 op/s 2024-03-24T11:47:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:25 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/694227345' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:25.415 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:47:25.415 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:47:25.415 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:47:25.415 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-03-24T11:47:25.416 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "", 2024-03-24T11:47:25.416 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 image" 2024-03-24T11:47:25.416 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:47:25.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 19655 19.70 MB/sec warmup 31 sec latency 75.721 ms 2024-03-24T11:47:26.077 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:47:26.290 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:26 smithi032 ceph-mon[24793]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:26.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:26 smithi146 ceph-mon[26954]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:26.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 20154 19.30 MB/sec warmup 32 sec latency 32.987 ms 2024-03-24T11:47:27.050 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:27 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1300038643' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:27.050 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:27 smithi032 ceph-mon[24793]: pgmap v252: 97 pgs: 97 active+clean; 177 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.3 MiB/s wr, 853 op/s 2024-03-24T11:47:27.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:27 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1300038643' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:27.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:27 smithi146 ceph-mon[26954]: pgmap v252: 97 pgs: 97 active+clean; 177 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.3 MiB/s wr, 853 op/s 2024-03-24T11:47:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 21054 19.85 MB/sec warmup 33 sec latency 19.926 ms 2024-03-24T11:47:28.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 21671 19.93 MB/sec warmup 34 sec latency 16.431 ms 2024-03-24T11:47:28.818 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:28 smithi032 ceph-mon[24793]: pgmap v253: 97 pgs: 97 active+clean; 174 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.8 MiB/s wr, 784 op/s 2024-03-24T11:47:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:28 smithi146 ceph-mon[26954]: pgmap v253: 97 pgs: 97 active+clean; 174 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.8 MiB/s wr, 784 op/s 2024-03-24T11:47:29.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 22180 19.49 MB/sec warmup 35 sec latency 24.444 ms 2024-03-24T11:47:30.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 22770 19.48 MB/sec warmup 36 sec latency 56.649 ms 2024-03-24T11:47:30.819 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:30 smithi032 ceph-mon[24793]: pgmap v254: 97 pgs: 97 active+clean; 181 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 685 op/s 2024-03-24T11:47:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:30 smithi146 ceph-mon[26954]: pgmap v254: 97 pgs: 97 active+clean; 181 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 685 op/s 2024-03-24T11:47:31.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 23521 19.20 MB/sec warmup 37 sec latency 40.979 ms 2024-03-24T11:47:32.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 24028 19.16 MB/sec warmup 38 sec latency 85.498 ms 2024-03-24T11:47:33.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:33 smithi032 ceph-mon[24793]: pgmap v255: 97 pgs: 97 active+clean; 191 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 7.8 KiB/s rd, 8.6 MiB/s wr, 908 op/s 2024-03-24T11:47:33.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:33 smithi146 ceph-mon[26954]: pgmap v255: 97 pgs: 97 active+clean; 191 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 7.8 KiB/s rd, 8.6 MiB/s wr, 908 op/s 2024-03-24T11:47:33.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 24605 19.25 MB/sec warmup 39 sec latency 32.406 ms 2024-03-24T11:47:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 25098 19.29 MB/sec warmup 40 sec latency 37.743 ms 2024-03-24T11:47:34.879 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:34 smithi032 ceph-mon[24793]: pgmap v256: 97 pgs: 97 active+clean; 191 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 697 op/s 2024-03-24T11:47:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:34 smithi146 ceph-mon[26954]: pgmap v256: 97 pgs: 97 active+clean; 191 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 697 op/s 2024-03-24T11:47:35.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 25470 18.90 MB/sec warmup 41 sec latency 30.635 ms 2024-03-24T11:47:36.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 25888 18.60 MB/sec warmup 42 sec latency 68.440 ms 2024-03-24T11:47:36.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:36 smithi032 ceph-mon[24793]: pgmap v257: 97 pgs: 97 active+clean; 197 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.6 MiB/s wr, 711 op/s 2024-03-24T11:47:37.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:36 smithi146 ceph-mon[26954]: pgmap v257: 97 pgs: 97 active+clean; 197 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.6 MiB/s wr, 711 op/s 2024-03-24T11:47:37.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 26386 18.54 MB/sec warmup 43 sec latency 76.228 ms 2024-03-24T11:47:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 27122 18.33 MB/sec warmup 44 sec latency 20.629 ms 2024-03-24T11:47:38.947 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:38 smithi032 ceph-mon[24793]: pgmap v258: 97 pgs: 97 active+clean; 203 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 8.1 MiB/s wr, 687 op/s 2024-03-24T11:47:39.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:38 smithi146 ceph-mon[26954]: pgmap v258: 97 pgs: 97 active+clean; 203 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 8.6 KiB/s rd, 8.1 MiB/s wr, 687 op/s 2024-03-24T11:47:39.432 INFO:teuthology.orchestra.run.smithi032.stdout: 5 27879 18.55 MB/sec warmup 45 sec latency 19.277 ms 2024-03-24T11:47:40.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 28598 18.77 MB/sec warmup 46 sec latency 28.818 ms 2024-03-24T11:47:40.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:40 smithi032 ceph-mon[24793]: pgmap v259: 97 pgs: 97 active+clean; 208 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 773 op/s 2024-03-24T11:47:41.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:40 smithi146 ceph-mon[26954]: pgmap v259: 97 pgs: 97 active+clean; 208 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 773 op/s 2024-03-24T11:47:41.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 29142 18.55 MB/sec warmup 47 sec latency 24.231 ms 2024-03-24T11:47:42.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 29486 18.29 MB/sec warmup 48 sec latency 33.860 ms 2024-03-24T11:47:43.246 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:43 smithi032 ceph-mon[24793]: pgmap v260: 97 pgs: 97 active+clean; 215 MiB data, 898 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.3 MiB/s wr, 910 op/s 2024-03-24T11:47:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:43 smithi146 ceph-mon[26954]: pgmap v260: 97 pgs: 97 active+clean; 215 MiB data, 898 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.3 MiB/s wr, 910 op/s 2024-03-24T11:47:43.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 30018 18.22 MB/sec warmup 49 sec latency 35.024 ms 2024-03-24T11:47:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 30580 18.02 MB/sec warmup 50 sec latency 38.703 ms 2024-03-24T11:47:45.034 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:44 smithi032 ceph-mon[24793]: pgmap v261: 97 pgs: 97 active+clean; 214 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 687 op/s 2024-03-24T11:47:45.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:44 smithi146 ceph-mon[26954]: pgmap v261: 97 pgs: 97 active+clean; 214 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 687 op/s 2024-03-24T11:47:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 31209 18.12 MB/sec warmup 51 sec latency 25.018 ms 2024-03-24T11:47:46.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 31862 18.27 MB/sec warmup 52 sec latency 22.958 ms 2024-03-24T11:47:47.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:46 smithi032 ceph-mon[24793]: pgmap v262: 97 pgs: 97 active+clean; 221 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 852 op/s 2024-03-24T11:47:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:46 smithi146 ceph-mon[26954]: pgmap v262: 97 pgs: 97 active+clean; 221 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 852 op/s 2024-03-24T11:47:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 32259 18.18 MB/sec warmup 53 sec latency 30.892 ms 2024-03-24T11:47:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 32726 17.92 MB/sec warmup 54 sec latency 32.035 ms 2024-03-24T11:47:49.016 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:48 smithi032 ceph-mon[24793]: pgmap v263: 97 pgs: 97 active+clean; 219 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 784 op/s 2024-03-24T11:47:49.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:48 smithi146 ceph-mon[26954]: pgmap v263: 97 pgs: 97 active+clean; 219 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 784 op/s 2024-03-24T11:47:49.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 33207 17.82 MB/sec warmup 55 sec latency 59.474 ms 2024-03-24T11:47:50.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 33869 17.68 MB/sec warmup 56 sec latency 71.309 ms 2024-03-24T11:47:50.792 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:50 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:50.792 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:50 smithi032 ceph-mon[24793]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T11:47:50.792 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:50 smithi032 ceph-mon[24793]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T11:47:50.792 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:50 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:50.792 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:50 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:50.792 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:50 smithi032 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.smithi032.xjdgbw) 2024-03-24T11:47:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:50 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:50 smithi146 ceph-mon[26954]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T11:47:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:50 smithi146 ceph-mon[26954]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T11:47:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:50 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:50 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:47:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:50 smithi146 ceph-mon[26954]: Upgrade: Need to upgrade myself (mgr.smithi032.xjdgbw) 2024-03-24T11:47:51.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 34559 17.72 MB/sec warmup 57 sec latency 30.830 ms 2024-03-24T11:47:51.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:51 smithi032 ceph-mon[24793]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 on smithi146 2024-03-24T11:47:51.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:51 smithi032 ceph-mon[24793]: pgmap v264: 97 pgs: 97 active+clean; 220 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 731 op/s 2024-03-24T11:47:52.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:51 smithi146 ceph-mon[26954]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 on smithi146 2024-03-24T11:47:52.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:51 smithi146 ceph-mon[26954]: pgmap v264: 97 pgs: 97 active+clean; 220 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 731 op/s 2024-03-24T11:47:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 35217 17.87 MB/sec warmup 58 sec latency 35.161 ms 2024-03-24T11:47:53.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:53 smithi032 ceph-mon[24793]: pgmap v265: 97 pgs: 97 active+clean; 225 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.5 MiB/s wr, 842 op/s 2024-03-24T11:47:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:53 smithi146 ceph-mon[26954]: pgmap v265: 97 pgs: 97 active+clean; 225 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.5 MiB/s wr, 842 op/s 2024-03-24T11:47:53.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 35695 17.88 MB/sec warmup 59 sec latency 32.896 ms 2024-03-24T11:47:54.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 36095 17.69 MB/sec warmup 60 sec latency 52.139 ms 2024-03-24T11:47:54.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:54 smithi032 ceph-mon[24793]: pgmap v266: 97 pgs: 97 active+clean; 226 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-03-24T11:47:55.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:54 smithi146 ceph-mon[26954]: pgmap v266: 97 pgs: 97 active+clean; 226 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-03-24T11:47:55.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 36541 17.54 MB/sec warmup 61 sec latency 590.309 ms 2024-03-24T11:47:56.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 37306 17.55 MB/sec warmup 62 sec latency 38.581 ms 2024-03-24T11:47:56.636 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:47:56.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:56 smithi032 ceph-mon[24793]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:56.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:47:56.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:56 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:47:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:56 smithi146 ceph-mon[26954]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:47:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:56 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:47:57.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 37941 17.41 MB/sec warmup 63 sec latency 34.940 ms 2024-03-24T11:47:57.625 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:47:57.625 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 66s ago 7m 22.8M - ba2b418f427c 34e04a907cd2 2024-03-24T11:47:57.625 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 66s ago 7m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (6m) 67s ago 6m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 66s ago 7m 53.4M - 8.3.5 dad864ee21e9 3669a5af1bd5 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (97s) 66s ago 96s 15.6M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (94s) 67s ago 94s 14.2M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:9283 running (8m) 66s ago 8m 461M - 17.2.0 e1d6a67b021e 7b699f0e61e8 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (6m) 67s ago 6m 414M - 17.2.0 e1d6a67b021e 546cb874d84e 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 66s ago 8m 46.5M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (6m) 67s ago 6m 37.9M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (76s) 66s ago 76s 46.9M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (71s) 67s ago 70s 52.9M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 66s ago 7m 19.6M - 1dbe0e931976 8bec60c4b714 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (6m) 67s ago 6m 16.0M - 1dbe0e931976 9c0cab415cfb 2024-03-24T11:47:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 66s ago 6m 46.1M 3217M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 66s ago 5m 42.8M 3217M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 66s ago 5m 42.7M 3217M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (4m) 66s ago 4m 43.4M 3217M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (4m) 67s ago 4m 39.2M 3984M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (3m) 67s ago 3m 39.3M 3984M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (3m) 67s ago 3m 41.4M 3984M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (3m) 67s ago 3m 43.3M 3984M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:47:57.627 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 66s ago 6m 62.4M - 514e6a882f6e 5c09869266dc 2024-03-24T11:47:58.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:58 smithi032 ceph-mon[24793]: pgmap v267: 97 pgs: 97 active+clean; 239 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 8.2 MiB/s wr, 813 op/s 2024-03-24T11:47:58.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:58 smithi032 ceph-mon[24793]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:58.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:58 smithi032 ceph-mon[24793]: from='client.24399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:58.194 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:47:58.195 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-24T11:47:58.196 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:47:58.196 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:47:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:58 smithi146 ceph-mon[26954]: pgmap v267: 97 pgs: 97 active+clean; 239 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 8.2 MiB/s wr, 813 op/s 2024-03-24T11:47:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:58 smithi146 ceph-mon[26954]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:58 smithi146 ceph-mon[26954]: from='client.24399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:58.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 38589 17.58 MB/sec warmup 64 sec latency 52.409 ms 2024-03-24T11:47:58.690 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:47:58.690 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:47:58.690 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:47:58.690 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-03-24T11:47:58.690 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "0/23 daemons upgraded", 2024-03-24T11:47:58.691 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 image on host smithi146" 2024-03-24T11:47:58.691 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:47:59.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:59 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3042990209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:59.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:47:59 smithi032 ceph-mon[24793]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:59.326 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:47:59.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:59 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3042990209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:59.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:47:59 smithi146 ceph-mon[26954]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:59.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 39262 17.76 MB/sec warmup 65 sec latency 22.266 ms 2024-03-24T11:48:00.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:00 smithi032 ceph-mon[24793]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.5 MiB/s wr, 650 op/s 2024-03-24T11:48:00.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:00 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3145276063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:48:00.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:00 smithi146 ceph-mon[26954]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.5 MiB/s wr, 650 op/s 2024-03-24T11:48:00.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:00 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3145276063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:48:00.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 39750 17.54 MB/sec warmup 66 sec latency 46.876 ms 2024-03-24T11:48:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 40310 17.42 MB/sec warmup 67 sec latency 28.868 ms 2024-03-24T11:48:02.144 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:02 smithi032 ceph-mon[24793]: pgmap v269: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 853 op/s 2024-03-24T11:48:02.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:02 smithi146 ceph-mon[26954]: pgmap v269: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 853 op/s 2024-03-24T11:48:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 41166 17.43 MB/sec warmup 68 sec latency 25.078 ms 2024-03-24T11:48:03.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 41608 17.34 MB/sec warmup 69 sec latency 52.201 ms 2024-03-24T11:48:04.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:04 smithi032 ceph-mon[24793]: pgmap v270: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 852 op/s 2024-03-24T11:48:04.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:04 smithi146 ceph-mon[26954]: pgmap v270: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 852 op/s 2024-03-24T11:48:04.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 42331 17.54 MB/sec warmup 70 sec latency 29.558 ms 2024-03-24T11:48:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 42796 17.60 MB/sec warmup 71 sec latency 48.078 ms 2024-03-24T11:48:06.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:06 smithi032 ceph-mon[24793]: pgmap v271: 97 pgs: 97 active+clean; 256 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.3 MiB/s wr, 785 op/s 2024-03-24T11:48:06.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:06 smithi146 ceph-mon[26954]: pgmap v271: 97 pgs: 97 active+clean; 256 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.3 MiB/s wr, 785 op/s 2024-03-24T11:48:06.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 42986 17.39 MB/sec warmup 72 sec latency 124.070 ms 2024-03-24T11:48:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 43422 17.19 MB/sec warmup 73 sec latency 103.091 ms 2024-03-24T11:48:08.085 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:08 smithi032 ceph-mon[24793]: pgmap v272: 97 pgs: 97 active+clean; 256 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 900 op/s 2024-03-24T11:48:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:08 smithi146 ceph-mon[26954]: pgmap v272: 97 pgs: 97 active+clean; 256 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 900 op/s 2024-03-24T11:48:08.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 43849 17.07 MB/sec warmup 74 sec latency 19.227 ms 2024-03-24T11:48:09.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 44846 17.10 MB/sec warmup 75 sec latency 26.193 ms 2024-03-24T11:48:10.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:10 smithi032 ceph-mon[24793]: pgmap v273: 97 pgs: 97 active+clean; 265 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 840 op/s 2024-03-24T11:48:10.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:10 smithi146 ceph-mon[26954]: pgmap v273: 97 pgs: 97 active+clean; 265 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 840 op/s 2024-03-24T11:48:10.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 45872 17.41 MB/sec warmup 76 sec latency 18.839 ms 2024-03-24T11:48:11.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 46549 17.50 MB/sec warmup 77 sec latency 22.895 ms 2024-03-24T11:48:12.105 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:12 smithi032 ceph-mon[24793]: pgmap v274: 97 pgs: 97 active+clean; 264 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 829 op/s 2024-03-24T11:48:12.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:12 smithi146 ceph-mon[26954]: pgmap v274: 97 pgs: 97 active+clean; 264 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 829 op/s 2024-03-24T11:48:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: 5 47280 17.41 MB/sec warmup 78 sec latency 21.780 ms 2024-03-24T11:48:13.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 48261 17.43 MB/sec warmup 79 sec latency 26.226 ms 2024-03-24T11:48:14.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:14 smithi032 ceph-mon[24793]: pgmap v275: 97 pgs: 97 active+clean; 271 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 778 op/s 2024-03-24T11:48:14.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:14 smithi146 ceph-mon[26954]: pgmap v275: 97 pgs: 97 active+clean; 271 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 778 op/s 2024-03-24T11:48:14.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 49002 17.57 MB/sec warmup 80 sec latency 24.168 ms 2024-03-24T11:48:15.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 49815 17.75 MB/sec warmup 81 sec latency 26.309 ms 2024-03-24T11:48:16.132 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:16 smithi032 ceph-mon[24793]: pgmap v276: 97 pgs: 97 active+clean; 279 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 885 op/s 2024-03-24T11:48:16.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:16 smithi146 ceph-mon[26954]: pgmap v276: 97 pgs: 97 active+clean; 279 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 885 op/s 2024-03-24T11:48:16.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 50431 17.62 MB/sec warmup 82 sec latency 13.990 ms 2024-03-24T11:48:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 51027 17.59 MB/sec warmup 83 sec latency 21.242 ms 2024-03-24T11:48:18.158 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:18 smithi032 ceph-mon[24793]: pgmap v277: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 888 op/s 2024-03-24T11:48:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:18 smithi146 ceph-mon[26954]: pgmap v277: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 888 op/s 2024-03-24T11:48:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 51871 17.54 MB/sec warmup 84 sec latency 27.034 ms 2024-03-24T11:48:19.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 52465 17.59 MB/sec warmup 85 sec latency 20.206 ms 2024-03-24T11:48:20.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:20 smithi032 ceph-mon[24793]: pgmap v278: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 894 op/s 2024-03-24T11:48:20.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:20 smithi146 ceph-mon[26954]: pgmap v278: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 894 op/s 2024-03-24T11:48:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 53183 17.73 MB/sec warmup 86 sec latency 67.772 ms 2024-03-24T11:48:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 53608 17.69 MB/sec warmup 87 sec latency 27.236 ms 2024-03-24T11:48:22.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:22 smithi032 ceph-mon[24793]: pgmap v279: 97 pgs: 97 active+clean; 290 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 890 op/s 2024-03-24T11:48:22.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:22 smithi146 ceph-mon[26954]: pgmap v279: 97 pgs: 97 active+clean; 290 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 890 op/s 2024-03-24T11:48:22.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 54123 17.53 MB/sec warmup 88 sec latency 36.426 ms 2024-03-24T11:48:23.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 54615 17.48 MB/sec warmup 89 sec latency 23.542 ms 2024-03-24T11:48:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:24 smithi146 ceph-mon[26954]: pgmap v280: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 999 op/s 2024-03-24T11:48:24.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:24 smithi032 ceph-mon[24793]: pgmap v280: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 999 op/s 2024-03-24T11:48:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 55360 17.43 MB/sec warmup 90 sec latency 34.765 ms 2024-03-24T11:48:25.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 55873 17.42 MB/sec warmup 91 sec latency 42.900 ms 2024-03-24T11:48:25.832 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:25 smithi032 ceph-mon[24793]: Upgrade: Updating mgr.smithi146.tfohfl 2024-03-24T11:48:25.832 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:25.832 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:25.832 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:25.832 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:25 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:25.832 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:25 smithi032 ceph-mon[24793]: Deploying daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:48:26.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:25 smithi146 ceph-mon[26954]: Upgrade: Updating mgr.smithi146.tfohfl 2024-03-24T11:48:26.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:26.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:26.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:26.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:25 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:26.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:25 smithi146 ceph-mon[26954]: Deploying daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:48:26.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 56631 17.57 MB/sec warmup 92 sec latency 67.302 ms 2024-03-24T11:48:26.873 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:26 smithi032 ceph-mon[24793]: pgmap v281: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 904 op/s 2024-03-24T11:48:27.024 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:26 smithi146 ceph-mon[26954]: pgmap v281: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 904 op/s 2024-03-24T11:48:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 57048 17.56 MB/sec warmup 93 sec latency 30.240 ms 2024-03-24T11:48:28.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 57318 17.39 MB/sec warmup 94 sec latency 133.498 ms 2024-03-24T11:48:28.906 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:28 smithi032 ceph-mon[24793]: pgmap v282: 97 pgs: 97 active+clean; 307 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.02k op/s 2024-03-24T11:48:29.086 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:28 smithi146 ceph-mon[26954]: pgmap v282: 97 pgs: 97 active+clean; 307 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.02k op/s 2024-03-24T11:48:29.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 57677 17.25 MB/sec warmup 95 sec latency 144.636 ms 2024-03-24T11:48:29.845 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:48:30.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 58161 17.23 MB/sec warmup 96 sec latency 90.174 ms 2024-03-24T11:48:30.825 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:48:30.825 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 99s ago 8m 22.8M - ba2b418f427c 34e04a907cd2 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 99s ago 7m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (7m) 101s ago 7m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 99s ago 7m 53.4M - 8.3.5 dad864ee21e9 3669a5af1bd5 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (2m) 99s ago 2m 15.6M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (2m) 101s ago 2m 14.2M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:9283 running (9m) 99s ago 9m 461M - 17.2.0 e1d6a67b021e 7b699f0e61e8 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (7m) 101s ago 7m 414M - 17.2.0 e1d6a67b021e 546cb874d84e 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 99s ago 9m 46.5M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (7m) 101s ago 7m 37.9M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (109s) 99s ago 109s 46.9M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (104s) 101s ago 103s 52.9M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 99s ago 7m 19.6M - 1dbe0e931976 8bec60c4b714 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (7m) 101s ago 7m 16.0M - 1dbe0e931976 9c0cab415cfb 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 99s ago 6m 46.1M 3217M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:48:30.826 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 99s ago 6m 42.8M 3217M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:48:30.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 99s ago 5m 42.7M 3217M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:48:30.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (5m) 99s ago 5m 43.4M 3217M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:48:30.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (4m) 101s ago 4m 39.2M 3984M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:48:30.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (4m) 101s ago 4m 39.3M 3984M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:48:30.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (4m) 101s ago 4m 41.4M 3984M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:48:30.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (3m) 101s ago 3m 43.3M 3984M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:48:30.827 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 99s ago 6m 62.4M - 514e6a882f6e 5c09869266dc 2024-03-24T11:48:30.920 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:30 smithi032 ceph-mon[24793]: pgmap v283: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 894 op/s 2024-03-24T11:48:31.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:30 smithi146 ceph-mon[26954]: pgmap v283: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 894 op/s 2024-03-24T11:48:31.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 58798 17.14 MB/sec warmup 97 sec latency 38.237 ms 2024-03-24T11:48:31.452 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:48:31.452 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:48:31.452 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:48:31.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:48:31.453 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:48:31.927 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:31 smithi032 ceph-mon[24793]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:31.927 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:31 smithi032 ceph-mon[24793]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:31.927 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:31 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3577115936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:31.930 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:48:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:48:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:48:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-03-24T11:48:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "0/23 daemons upgraded", 2024-03-24T11:48:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mgr daemons" 2024-03-24T11:48:31.930 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:48:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:31 smithi146 ceph-mon[26954]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:31 smithi146 ceph-mon[26954]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:32.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:31 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3577115936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:32.408 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:48:32.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 59256 17.06 MB/sec warmup 98 sec latency 62.244 ms 2024-03-24T11:48:32.939 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:32 smithi032 ceph-mon[24793]: pgmap v284: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.2 MiB/s wr, 763 op/s 2024-03-24T11:48:32.939 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:32 smithi032 ceph-mon[24793]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:32.939 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:32 smithi032 ceph-mon[24793]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:32.939 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:32 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/790069205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:48:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:32 smithi146 ceph-mon[26954]: pgmap v284: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.2 MiB/s wr, 763 op/s 2024-03-24T11:48:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:32 smithi146 ceph-mon[26954]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:32 smithi146 ceph-mon[26954]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:32 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/790069205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:48:33.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 59859 17.17 MB/sec warmup 99 sec latency 45.079 ms 2024-03-24T11:48:34.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:34 smithi032 ceph-mon[24793]: pgmap v285: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 771 op/s 2024-03-24T11:48:34.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:34 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:34.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:34 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:34.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:34 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:34.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:34 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:34 smithi146 ceph-mon[26954]: pgmap v285: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 771 op/s 2024-03-24T11:48:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:34 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:34 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:34 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:34.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:34 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 60416 17.24 MB/sec warmup 100 sec latency 32.229 ms 2024-03-24T11:48:35.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 60678 17.12 MB/sec warmup 101 sec latency 51.929 ms 2024-03-24T11:48:36.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:36 smithi032 ceph-mon[24793]: pgmap v286: 97 pgs: 97 active+clean; 308 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 751 op/s 2024-03-24T11:48:36.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:36 smithi146 ceph-mon[26954]: pgmap v286: 97 pgs: 97 active+clean; 308 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 7.2 MiB/s wr, 751 op/s 2024-03-24T11:48:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 61005 16.97 MB/sec warmup 102 sec latency 40.786 ms 2024-03-24T11:48:37.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 61435 16.88 MB/sec warmup 103 sec latency 38.622 ms 2024-03-24T11:48:38.048 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:38 smithi032 ceph-mon[24793]: pgmap v287: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.9 MiB/s wr, 847 op/s 2024-03-24T11:48:38.278 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:38 smithi146 ceph-mon[26954]: pgmap v287: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.9 MiB/s wr, 847 op/s 2024-03-24T11:48:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 62066 16.89 MB/sec warmup 104 sec latency 28.914 ms 2024-03-24T11:48:39.275 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:39 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:39.275 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:39 smithi146 ceph-mon[26954]: pgmap v288: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.9 MiB/s wr, 593 op/s 2024-03-24T11:48:39.275 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:39 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:39.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:39 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:39.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:39 smithi032 ceph-mon[24793]: pgmap v288: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.9 MiB/s wr, 593 op/s 2024-03-24T11:48:39.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:39 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:39.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 62668 16.80 MB/sec warmup 105 sec latency 34.429 ms 2024-03-24T11:48:40.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 63106 16.81 MB/sec warmup 106 sec latency 25.959 ms 2024-03-24T11:48:40.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:40 smithi032 ceph-mon[24793]: pgmap v289: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.7 MiB/s wr, 666 op/s 2024-03-24T11:48:41.106 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:40 smithi146 ceph-mon[26954]: pgmap v289: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.7 MiB/s wr, 666 op/s 2024-03-24T11:48:41.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 63652 16.85 MB/sec warmup 107 sec latency 59.061 ms 2024-03-24T11:48:42.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 64137 16.89 MB/sec warmup 108 sec latency 23.767 ms 2024-03-24T11:48:43.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:43 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:48:43.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:43 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:43.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:43 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:48:43.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:43 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:43.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:43 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:48:43.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:43 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:48:43.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:43 smithi032 ceph-mon[24793]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.6 MiB/s wr, 805 op/s 2024-03-24T11:48:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:43 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:48:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:43 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:43 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:48:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:43 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:43 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:48:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:43 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:48:43.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:43 smithi146 ceph-mon[26954]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.6 MiB/s wr, 805 op/s 2024-03-24T11:48:43.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 64537 16.76 MB/sec warmup 109 sec latency 26.584 ms 2024-03-24T11:48:44.140 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: mgrmap e22: smithi032.xjdgbw(active, since 7m), standbys: smithi146.tfohfl 2024-03-24T11:48:44.140 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:44.140 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:44.140 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:44.140 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:44.141 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.smithi032.xjdgbw) 2024-03-24T11:48:44.141 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: Failing over to other MGR 2024-03-24T11:48:44.141 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr fail", "who": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:48:44.141 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:48:44.141 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:44 smithi032 ceph-mon[24793]: Standby manager daemon smithi032.xjdgbw started 2024-03-24T11:48:44.318 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: mgrmap e22: smithi032.xjdgbw(active, since 7m), standbys: smithi146.tfohfl 2024-03-24T11:48:44.318 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:44.318 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:44.318 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:48:44.318 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:44.319 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: Upgrade: Need to upgrade myself (mgr.smithi032.xjdgbw) 2024-03-24T11:48:44.319 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: Failing over to other MGR 2024-03-24T11:48:44.319 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr fail", "who": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:48:44.319 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:48:44.319 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:44 smithi146 ceph-mon[26954]: Standby manager daemon smithi032.xjdgbw started 2024-03-24T11:48:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 64953 16.67 MB/sec warmup 110 sec latency 30.119 ms 2024-03-24T11:48:45.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:48:45.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:48:45.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:48:45.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:48:45.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "mgr fail", "who": "smithi032.xjdgbw"}]': finished 2024-03-24T11:48:45.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: mgrmap e23: smithi146.tfohfl(active, starting, since 1.00342s), standbys: smithi032.xjdgbw 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr metadata", "who": "smithi146.tfohfl", "id": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:48:45.406 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: Manager daemon smithi146.tfohfl is now available 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/trash_purge_schedule"}]: dispatch 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/trash_purge_schedule"}]: dispatch 2024-03-24T11:48:45.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:45 smithi032 ceph-mon[24793]: Deploying cephadm binary to smithi146 2024-03-24T11:48:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: 5 65407 16.66 MB/sec warmup 111 sec latency 60.938 ms 2024-03-24T11:48:45.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:48:45.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:48:45.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.14202 172.21.15.32:0/3570634920' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "mgr fail", "who": "smithi032.xjdgbw"}]': finished 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: mgrmap e23: smithi146.tfohfl(active, starting, since 1.00342s), standbys: smithi032.xjdgbw 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr metadata", "who": "smithi146.tfohfl", "id": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:48:45.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: Manager daemon smithi146.tfohfl is now available 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/trash_purge_schedule"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi146.tfohfl/trash_purge_schedule"}]: dispatch 2024-03-24T11:48:45.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:45 smithi146 ceph-mon[26954]: Deploying cephadm binary to smithi146 2024-03-24T11:48:46.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:46 smithi032 ceph-mon[24793]: Deploying cephadm binary to smithi032 2024-03-24T11:48:46.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:46 smithi032 ceph-mon[24793]: mgrmap e24: smithi146.tfohfl(active, since 2s), standbys: smithi032.xjdgbw 2024-03-24T11:48:46.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:46 smithi032 ceph-mon[24793]: pgmap v3: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:46.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:46 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:48:46.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:46 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:48:46.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 65899 16.56 MB/sec warmup 112 sec latency 92.758 ms 2024-03-24T11:48:46.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:46 smithi146 ceph-mon[26954]: Deploying cephadm binary to smithi032 2024-03-24T11:48:46.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:46 smithi146 ceph-mon[26954]: mgrmap e24: smithi146.tfohfl(active, since 2s), standbys: smithi032.xjdgbw 2024-03-24T11:48:46.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:46 smithi146 ceph-mon[26954]: pgmap v3: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:46.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:46 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:48:46.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:46 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:48:47.382 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:47 smithi032 ceph-mon[24793]: pgmap v4: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 66315 16.48 MB/sec warmup 113 sec latency 33.753 ms 2024-03-24T11:48:47.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:47 smithi146 ceph-mon[26954]: pgmap v4: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:48.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:48 smithi032 ceph-mon[24793]: [24/Mar/2024:11:48:47] ENGINE Bus STARTING 2024-03-24T11:48:48.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:48 smithi032 ceph-mon[24793]: mgrmap e25: smithi146.tfohfl(active, since 4s), standbys: smithi032.xjdgbw 2024-03-24T11:48:48.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:48 smithi032 ceph-mon[24793]: [24/Mar/2024:11:48:47] ENGINE Serving on https://172.21.15.146:7150 2024-03-24T11:48:48.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:48 smithi032 ceph-mon[24793]: [24/Mar/2024:11:48:47] ENGINE Serving on http://172.21.15.146:8765 2024-03-24T11:48:48.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:48 smithi032 ceph-mon[24793]: [24/Mar/2024:11:48:47] ENGINE Bus STARTED 2024-03-24T11:48:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 66715 16.50 MB/sec warmup 114 sec latency 34.740 ms 2024-03-24T11:48:48.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:48 smithi146 ceph-mon[26954]: [24/Mar/2024:11:48:47] ENGINE Bus STARTING 2024-03-24T11:48:48.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:48 smithi146 ceph-mon[26954]: mgrmap e25: smithi146.tfohfl(active, since 4s), standbys: smithi032.xjdgbw 2024-03-24T11:48:48.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:48 smithi146 ceph-mon[26954]: [24/Mar/2024:11:48:47] ENGINE Serving on https://172.21.15.146:7150 2024-03-24T11:48:48.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:48 smithi146 ceph-mon[26954]: [24/Mar/2024:11:48:47] ENGINE Serving on http://172.21.15.146:8765 2024-03-24T11:48:48.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:48 smithi146 ceph-mon[26954]: [24/Mar/2024:11:48:47] ENGINE Bus STARTED 2024-03-24T11:48:49.427 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:49 smithi032 ceph-mon[24793]: pgmap v5: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:49.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 67173 16.50 MB/sec warmup 115 sec latency 39.595 ms 2024-03-24T11:48:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:49 smithi146 ceph-mon[26954]: pgmap v5: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:50.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 67593 16.52 MB/sec warmup 116 sec latency 70.555 ms 2024-03-24T11:48:51.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 67797 16.41 MB/sec warmup 117 sec latency 64.843 ms 2024-03-24T11:48:51.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:51 smithi032 ceph-mon[24793]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:51.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:51 smithi146 ceph-mon[26954]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T11:48:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 68195 16.30 MB/sec warmup 118 sec latency 30.900 ms 2024-03-24T11:48:53.155 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:53 smithi032 ceph-mon[24793]: pgmap v7: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 4.7 MiB/s wr, 519 op/s 2024-03-24T11:48:53.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:53 smithi032 ceph-mon[24793]: Standby manager daemon smithi032.xjdgbw restarted 2024-03-24T11:48:53.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:53 smithi032 ceph-mon[24793]: Standby manager daemon smithi032.xjdgbw started 2024-03-24T11:48:53.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:53 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/crt"}]: dispatch 2024-03-24T11:48:53.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:53 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:53.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:53 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/key"}]: dispatch 2024-03-24T11:48:53.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:53 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:53 smithi146 ceph-mon[26954]: pgmap v7: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 4.7 MiB/s wr, 519 op/s 2024-03-24T11:48:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:53 smithi146 ceph-mon[26954]: Standby manager daemon smithi032.xjdgbw restarted 2024-03-24T11:48:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:53 smithi146 ceph-mon[26954]: Standby manager daemon smithi032.xjdgbw started 2024-03-24T11:48:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/crt"}]: dispatch 2024-03-24T11:48:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/key"}]: dispatch 2024-03-24T11:48:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/3170812547' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:53.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 68761 16.29 MB/sec warmup 119 sec latency 32.129 ms 2024-03-24T11:48:54.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:54 smithi032 ceph-mon[24793]: mgrmap e26: smithi146.tfohfl(active, since 9s), standbys: smithi032.xjdgbw 2024-03-24T11:48:54.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:54 smithi146 ceph-mon[26954]: mgrmap e26: smithi146.tfohfl(active, since 9s), standbys: smithi032.xjdgbw 2024-03-24T11:48:55.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:55 smithi032 ceph-mon[24793]: pgmap v8: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.7 MiB/s wr, 403 op/s 2024-03-24T11:48:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:55 smithi146 ceph-mon[26954]: pgmap v8: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.7 MiB/s wr, 403 op/s 2024-03-24T11:48:55.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 70206 20.24 MB/sec execute 1 sec latency 44.677 ms 2024-03-24T11:48:56.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 70929 24.95 MB/sec execute 2 sec latency 78.060 ms 2024-03-24T11:48:57.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:57 smithi032 ceph-mon[24793]: pgmap v9: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.1 MiB/s wr, 678 op/s 2024-03-24T11:48:57.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 71379 21.69 MB/sec execute 3 sec latency 46.846 ms 2024-03-24T11:48:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:57 smithi146 ceph-mon[26954]: pgmap v9: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.1 MiB/s wr, 678 op/s 2024-03-24T11:48:58.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 71783 17.26 MB/sec execute 4 sec latency 32.179 ms 2024-03-24T11:48:59.388 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:48:59 smithi032 ceph-mon[24793]: pgmap v10: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.6 MiB/s wr, 617 op/s 2024-03-24T11:48:59.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 72125 15.32 MB/sec execute 5 sec latency 37.380 ms 2024-03-24T11:48:59.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:48:59 smithi146 ceph-mon[26954]: pgmap v10: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.6 MiB/s wr, 617 op/s 2024-03-24T11:49:00.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:00 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:00.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:00 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:00.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:00 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:00.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:00 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:00.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:00 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:00.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 72758 15.45 MB/sec execute 6 sec latency 33.436 ms 2024-03-24T11:49:00.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:00 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:00.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:00 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:00.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:00 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:00.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:00 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:00.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:00 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 73490 15.24 MB/sec execute 7 sec latency 23.899 ms 2024-03-24T11:49:01.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: pgmap v11: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.6 MiB/s wr, 617 op/s 2024-03-24T11:49:01.905 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.905 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.906 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:01 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: pgmap v11: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.6 MiB/s wr, 617 op/s 2024-03-24T11:49:01.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:01.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:01.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:01 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 74208 16.92 MB/sec execute 8 sec latency 26.562 ms 2024-03-24T11:49:02.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:02 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi032 to 1937M 2024-03-24T11:49:02.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:02 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi032 to 1937M 2024-03-24T11:49:03.015 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:49:03.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 74656 17.05 MB/sec execute 9 sec latency 27.711 ms 2024-03-24T11:49:03.825 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:03 smithi146 ceph-mon[26954]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 921 op/s 2024-03-24T11:49:03.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:03 smithi032 ceph-mon[24793]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 921 op/s 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 4s ago 8m 26.4M - 0.23.0 ba2b418f427c 34e04a907cd2 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 4s ago 8m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (7m) 4s ago 7m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 4s ago 8m 53.5M - 8.3.5 dad864ee21e9 3669a5af1bd5 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (2m) 4s ago 2m 43.4M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (2m) 4s ago 2m 15.0M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:49:04.101 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:9283 running (9m) 4s ago 9m 386M - 17.2.0 e1d6a67b021e 7b699f0e61e8 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (30s) 4s ago 7m 492M - 19.0.0-1667-gdb0330b1 4f325f8440e9 eced178a5b22 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 4s ago 9m 51.4M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (7m) 4s ago 7m 41.1M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (2m) 4s ago 2m 243M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (2m) 4s ago 2m 78.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 4s ago 8m 21.4M - 1.3.1 1dbe0e931976 8bec60c4b714 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (7m) 4s ago 7m 16.0M - 1.3.1 1dbe0e931976 9c0cab415cfb 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 4s ago 7m 287M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 4s ago 6m 222M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 4s ago 6m 166M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (5m) 4s ago 5m 238M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (5m) 4s ago 5m 265M 3984M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:49:04.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (4m) 4s ago 4m 289M 3984M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:49:04.103 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (4m) 4s ago 4m 262M 3984M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:49:04.103 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (4m) 4s ago 4m 367M 3984M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:49:04.103 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 4s ago 7m 64.6M - 2.33.4 514e6a882f6e 5c09869266dc 2024-03-24T11:49:04.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 75031 16.14 MB/sec execute 10 sec latency 46.954 ms 2024-03-24T11:49:04.748 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:49:04.749 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-24T11:49:04.750 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:49:04.750 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:49:04.750 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:49:04.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:04 smithi146 ceph-mon[26954]: from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:04.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:04 smithi146 ceph-mon[26954]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:04.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:04 smithi032 ceph-mon[24793]: from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:04.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:04 smithi032 ceph-mon[24793]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:05.327 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:49:05.328 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:49:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 75586 15.49 MB/sec execute 11 sec latency 30.236 ms 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 621 op/s 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/254305468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:05.945 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:05 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:49:06.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:06.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 621 op/s 2024-03-24T11:49:06.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/254305468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:49:06.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:06.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:05 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:06.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 76259 15.72 MB/sec execute 12 sec latency 26.448 ms 2024-03-24T11:49:06.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: Adjusting osd_memory_target on smithi146 to 2704M 2024-03-24T11:49:06.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:49:06.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:49:06.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:06.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:49:06.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:49:06.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/336340183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:06.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:06 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: Adjusting osd_memory_target on smithi146 to 2704M 2024-03-24T11:49:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:49:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:49:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:49:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/336340183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:06 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 77058 15.48 MB/sec execute 13 sec latency 24.575 ms 2024-03-24T11:49:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:07 smithi032 ceph-mon[24793]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:49:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:07 smithi032 ceph-mon[24793]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:49:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:07 smithi032 ceph-mon[24793]: pgmap v14: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 945 op/s 2024-03-24T11:49:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:07 smithi032 ceph-mon[24793]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-03-24T11:49:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:07 smithi032 ceph-mon[24793]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-03-24T11:49:08.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:07 smithi146 ceph-mon[26954]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:49:08.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:07 smithi146 ceph-mon[26954]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:49:08.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:07 smithi146 ceph-mon[26954]: pgmap v14: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 945 op/s 2024-03-24T11:49:08.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:07 smithi146 ceph-mon[26954]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-03-24T11:49:08.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:07 smithi146 ceph-mon[26954]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-03-24T11:49:08.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 77777 16.55 MB/sec execute 14 sec latency 43.842 ms 2024-03-24T11:49:09.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 78253 16.63 MB/sec execute 15 sec latency 93.104 ms 2024-03-24T11:49:09.786 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:09 smithi032 ceph-mon[24793]: pgmap v15: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.9 MiB/s wr, 627 op/s 2024-03-24T11:49:10.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:09 smithi146 ceph-mon[26954]: pgmap v15: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.9 MiB/s wr, 627 op/s 2024-03-24T11:49:10.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 78677 16.17 MB/sec execute 16 sec latency 28.323 ms 2024-03-24T11:49:11.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 79090 15.62 MB/sec execute 17 sec latency 29.108 ms 2024-03-24T11:49:11.805 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:11 smithi032 ceph-mon[24793]: pgmap v16: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.9 MiB/s wr, 627 op/s 2024-03-24T11:49:12.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:11 smithi146 ceph-mon[26954]: pgmap v16: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.9 MiB/s wr, 627 op/s 2024-03-24T11:49:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 79546 15.23 MB/sec execute 18 sec latency 54.245 ms 2024-03-24T11:49:13.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:13 smithi032 ceph-mon[24793]: pgmap v17: 97 pgs: 97 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 987 op/s 2024-03-24T11:49:13.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:13 smithi146 ceph-mon[26954]: pgmap v17: 97 pgs: 97 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 987 op/s 2024-03-24T11:49:13.435 INFO:teuthology.orchestra.run.smithi032.stdout: 5 80145 15.16 MB/sec execute 19 sec latency 37.282 ms 2024-03-24T11:49:14.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 80743 15.18 MB/sec execute 20 sec latency 27.100 ms 2024-03-24T11:49:15.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:15 smithi032 ceph-mon[24793]: pgmap v18: 97 pgs: 97 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.5 MiB/s wr, 683 op/s 2024-03-24T11:49:15.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:15 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:15.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 81248 15.50 MB/sec execute 21 sec latency 27.989 ms 2024-03-24T11:49:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:15 smithi146 ceph-mon[26954]: pgmap v18: 97 pgs: 97 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.5 MiB/s wr, 683 op/s 2024-03-24T11:49:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:15 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:16.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 81735 15.53 MB/sec execute 22 sec latency 32.321 ms 2024-03-24T11:49:17.389 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:17 smithi032 ceph-mon[24793]: pgmap v19: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 987 op/s 2024-03-24T11:49:17.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 82192 15.37 MB/sec execute 23 sec latency 62.328 ms 2024-03-24T11:49:17.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:17 smithi146 ceph-mon[26954]: pgmap v19: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 987 op/s 2024-03-24T11:49:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 82698 15.07 MB/sec execute 24 sec latency 24.489 ms 2024-03-24T11:49:19.403 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:19 smithi032 ceph-mon[24793]: pgmap v20: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-03-24T11:49:19.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 83225 14.94 MB/sec execute 25 sec latency 109.589 ms 2024-03-24T11:49:19.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:19 smithi146 ceph-mon[26954]: pgmap v20: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-03-24T11:49:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: 5 83883 14.90 MB/sec execute 26 sec latency 41.949 ms 2024-03-24T11:49:21.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:21 smithi032 ceph-mon[24793]: pgmap v21: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-03-24T11:49:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 84384 14.87 MB/sec execute 27 sec latency 27.768 ms 2024-03-24T11:49:21.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:21 smithi146 ceph-mon[26954]: pgmap v21: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-03-24T11:49:22.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 84960 15.30 MB/sec execute 28 sec latency 35.413 ms 2024-03-24T11:49:23.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:23 smithi032 ceph-mon[24793]: pgmap v22: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 978 op/s 2024-03-24T11:49:23.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:23 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:23.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:23 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:23.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:23 smithi146 ceph-mon[26954]: pgmap v22: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 978 op/s 2024-03-24T11:49:23.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:23 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:23.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:23 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:23.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 85457 15.34 MB/sec execute 29 sec latency 50.029 ms 2024-03-24T11:49:24.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:24 smithi032 ceph-mon[24793]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-03-24T11:49:24.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:24 smithi032 ceph-mon[24793]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:49:24.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:24 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:24.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:24 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:24.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:24 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:49:24.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:24 smithi032 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:49:24.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:24 smithi032 ceph-mon[24793]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-03-24T11:49:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:24 smithi146 ceph-mon[26954]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-03-24T11:49:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:24 smithi146 ceph-mon[26954]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:49:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:24 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:24 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:24 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:49:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:24 smithi146 ceph-mon[26954]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:49:24.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:24 smithi146 ceph-mon[26954]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-03-24T11:49:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 85781 15.00 MB/sec execute 30 sec latency 26.740 ms 2024-03-24T11:49:25.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:25 smithi032 ceph-mon[24793]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 618 op/s 2024-03-24T11:49:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:25 smithi146 ceph-mon[26954]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 618 op/s 2024-03-24T11:49:25.436 INFO:teuthology.orchestra.run.smithi032.stdout: 5 86244 14.76 MB/sec execute 31 sec latency 25.887 ms 2024-03-24T11:49:26.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 86643 14.63 MB/sec execute 32 sec latency 44.578 ms 2024-03-24T11:49:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: 5 87077 14.45 MB/sec execute 33 sec latency 43.974 ms 2024-03-24T11:49:27.527 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:27 smithi032 ceph-mon[24793]: pgmap v24: 97 pgs: 97 active+clean; 337 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 912 op/s 2024-03-24T11:49:27.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:27 smithi146 ceph-mon[26954]: pgmap v24: 97 pgs: 97 active+clean; 337 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 912 op/s 2024-03-24T11:49:28.437 INFO:teuthology.orchestra.run.smithi032.stdout: 5 87519 14.26 MB/sec execute 34 sec latency 34.218 ms 2024-03-24T11:49:29.303 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:29 smithi032 ceph-mon[24793]: pgmap v25: 97 pgs: 97 active+clean; 337 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.4 MiB/s wr, 609 op/s 2024-03-24T11:49:29.437 INFO:teuthology.orchestra.run.smithi032.stdout: 5 87895 14.19 MB/sec execute 35 sec latency 31.841 ms 2024-03-24T11:49:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:29 smithi146 ceph-mon[26954]: pgmap v25: 97 pgs: 97 active+clean; 337 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.4 MiB/s wr, 609 op/s 2024-03-24T11:49:30.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:30 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:30.437 INFO:teuthology.orchestra.run.smithi032.stdout: 5 88352 14.34 MB/sec execute 36 sec latency 32.970 ms 2024-03-24T11:49:30.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:30 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:31.380 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:31 smithi032 ceph-mon[24793]: pgmap v26: 97 pgs: 97 active+clean; 337 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.4 MiB/s wr, 609 op/s 2024-03-24T11:49:31.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 88702 14.35 MB/sec execute 37 sec latency 49.165 ms 2024-03-24T11:49:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:31 smithi146 ceph-mon[26954]: pgmap v26: 97 pgs: 97 active+clean; 337 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.4 MiB/s wr, 609 op/s 2024-03-24T11:49:32.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 89090 14.27 MB/sec execute 38 sec latency 32.867 ms 2024-03-24T11:49:33.157 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:33 smithi032 ceph-mon[24793]: pgmap v27: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 858 op/s 2024-03-24T11:49:33.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:33 smithi146 ceph-mon[26954]: pgmap v27: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 858 op/s 2024-03-24T11:49:33.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 89407 14.01 MB/sec execute 39 sec latency 28.312 ms 2024-03-24T11:49:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 89821 13.83 MB/sec execute 40 sec latency 33.751 ms 2024-03-24T11:49:35.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 90285 13.76 MB/sec execute 41 sec latency 56.831 ms 2024-03-24T11:49:35.440 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:35 smithi032 ceph-mon[24793]: pgmap v28: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 543 op/s 2024-03-24T11:49:35.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:35 smithi146 ceph-mon[26954]: pgmap v28: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 543 op/s 2024-03-24T11:49:36.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 90758 13.68 MB/sec execute 42 sec latency 102.471 ms 2024-03-24T11:49:36.674 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:49:37.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 91361 13.70 MB/sec execute 43 sec latency 26.881 ms 2024-03-24T11:49:37.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:37 smithi032 ceph-mon[24793]: pgmap v29: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-03-24T11:49:37.471 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:37 smithi032 ceph-mon[24793]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:37.610 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:49:37.610 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 starting - - - - 2024-03-24T11:49:37.610 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 38s ago 9m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (8m) 37s ago 8m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 38s ago 8m 53.5M - 8.3.5 dad864ee21e9 3669a5af1bd5 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (3m) 38s ago 3m 43.4M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (3m) 37s ago 3m 15.0M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:9283 running (10m) 38s ago 10m 386M - 17.2.0 e1d6a67b021e 7b699f0e61e8 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (64s) 37s ago 8m 492M - 19.0.0-1667-gdb0330b1 4f325f8440e9 eced178a5b22 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 38s ago 10m 51.4M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (8m) 37s ago 8m 41.1M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (2m) 38s ago 2m 243M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (2m) 37s ago 2m 78.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 38s ago 8m 21.4M - 1.3.1 1dbe0e931976 8bec60c4b714 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (8m) 37s ago 8m 16.0M - 1.3.1 1dbe0e931976 9c0cab415cfb 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 38s ago 7m 287M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 38s ago 7m 222M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:49:37.611 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 38s ago 6m 166M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:49:37.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (6m) 38s ago 6m 238M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:49:37.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (5m) 37s ago 5m 265M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:49:37.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (5m) 37s ago 5m 289M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:49:37.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (5m) 37s ago 5m 262M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:49:37.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (4m) 37s ago 4m 367M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:49:37.612 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8m) 38s ago 8m 64.6M - 2.33.4 514e6a882f6e 5c09869266dc 2024-03-24T11:49:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:37 smithi146 ceph-mon[26954]: pgmap v29: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-03-24T11:49:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:37 smithi146 ceph-mon[26954]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:38.148 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:49:38.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:49:38.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:49:38.149 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:49:38.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-24T11:49:38.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:49:38.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:49:38.149 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:49:38.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 91802 13.81 MB/sec execute 44 sec latency 32.827 ms 2024-03-24T11:49:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:38 smithi032 ceph-mon[24793]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:38 smithi032 ceph-mon[24793]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:38 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/3102772037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:49:38.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:38 smithi146 ceph-mon[26954]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:38.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:38 smithi146 ceph-mon[26954]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:38.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:38 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/3102772037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:49:38.733 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:49:38.733 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:49:38.733 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:49:38.733 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:49:38.733 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-03-24T11:49:38.734 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T11:49:38.734 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-03-24T11:49:38.734 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:49:38.734 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:49:39.291 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:49:39.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 92291 13.93 MB/sec execute 45 sec latency 44.510 ms 2024-03-24T11:49:39.489 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:39 smithi032 ceph-mon[24793]: pgmap v30: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.9 MiB/s wr, 495 op/s 2024-03-24T11:49:39.489 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:39 smithi032 ceph-mon[24793]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:39 smithi146 ceph-mon[26954]: pgmap v30: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.9 MiB/s wr, 495 op/s 2024-03-24T11:49:39.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:39 smithi146 ceph-mon[26954]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:40.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 92661 13.88 MB/sec execute 46 sec latency 43.031 ms 2024-03-24T11:49:40.507 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:40 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1300815332' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:49:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:40 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1300815332' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:49:41.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 93044 13.67 MB/sec execute 47 sec latency 31.128 ms 2024-03-24T11:49:41.519 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:41 smithi032 ceph-mon[24793]: pgmap v31: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.9 MiB/s wr, 495 op/s 2024-03-24T11:49:41.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:41 smithi146 ceph-mon[26954]: pgmap v31: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.9 MiB/s wr, 495 op/s 2024-03-24T11:49:42.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 93422 13.58 MB/sec execute 48 sec latency 74.307 ms 2024-03-24T11:49:43.300 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:43 smithi032 ceph-mon[24793]: pgmap v32: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 773 op/s 2024-03-24T11:49:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:43 smithi146 ceph-mon[26954]: pgmap v32: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 773 op/s 2024-03-24T11:49:43.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 93811 13.41 MB/sec execute 49 sec latency 87.533 ms 2024-03-24T11:49:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 94288 13.40 MB/sec execute 50 sec latency 41.746 ms 2024-03-24T11:49:45.334 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:45 smithi032 ceph-mon[24793]: pgmap v33: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.6 MiB/s wr, 524 op/s 2024-03-24T11:49:45.334 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 94864 13.41 MB/sec execute 51 sec latency 28.712 ms 2024-03-24T11:49:45.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:45 smithi146 ceph-mon[26954]: pgmap v33: 97 pgs: 97 active+clean; 338 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.6 MiB/s wr, 524 op/s 2024-03-24T11:49:45.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:46.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 95349 13.57 MB/sec execute 52 sec latency 32.339 ms 2024-03-24T11:49:47.371 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:47 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:47.371 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:47 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:47.371 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:47 smithi032 ceph-mon[24793]: pgmap v34: 97 pgs: 97 active+clean; 333 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 800 op/s 2024-03-24T11:49:47.371 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:47 smithi032 ceph-mon[24793]: Reconfiguring node-exporter.smithi032 (dependencies changed)... 2024-03-24T11:49:47.371 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:47 smithi032 ceph-mon[24793]: Reconfiguring daemon node-exporter.smithi032 on smithi032 2024-03-24T11:49:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 95795 13.58 MB/sec execute 53 sec latency 40.056 ms 2024-03-24T11:49:47.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:47 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:47.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:47 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:47.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:47 smithi146 ceph-mon[26954]: pgmap v34: 97 pgs: 97 active+clean; 333 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 800 op/s 2024-03-24T11:49:47.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:47 smithi146 ceph-mon[26954]: Reconfiguring node-exporter.smithi032 (dependencies changed)... 2024-03-24T11:49:47.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:47 smithi146 ceph-mon[26954]: Reconfiguring daemon node-exporter.smithi032 on smithi032 2024-03-24T11:49:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 96200 13.58 MB/sec execute 54 sec latency 38.163 ms 2024-03-24T11:49:49.411 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:49 smithi032 ceph-mon[24793]: pgmap v35: 97 pgs: 97 active+clean; 333 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 553 op/s 2024-03-24T11:49:49.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 96471 13.41 MB/sec execute 55 sec latency 61.161 ms 2024-03-24T11:49:49.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:49 smithi146 ceph-mon[26954]: pgmap v35: 97 pgs: 97 active+clean; 333 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 553 op/s 2024-03-24T11:49:50.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 96845 13.28 MB/sec execute 56 sec latency 40.373 ms 2024-03-24T11:49:50.680 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:50 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:50.680 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:50 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:50.680 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:50 smithi032 ceph-mon[24793]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-03-24T11:49:50.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:50 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:50.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:50 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:49:50.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:50 smithi146 ceph-mon[26954]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-03-24T11:49:51.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 97196 13.17 MB/sec execute 57 sec latency 36.256 ms 2024-03-24T11:49:51.697 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:51 smithi032 ceph-mon[24793]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-03-24T11:49:51.698 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:51 smithi032 ceph-mon[24793]: pgmap v36: 97 pgs: 97 active+clean; 333 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 553 op/s 2024-03-24T11:49:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:51 smithi146 ceph-mon[26954]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-03-24T11:49:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:51 smithi146 ceph-mon[26954]: pgmap v36: 97 pgs: 97 active+clean; 333 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 553 op/s 2024-03-24T11:49:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 97694 13.18 MB/sec execute 58 sec latency 33.255 ms 2024-03-24T11:49:53.228 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:53 smithi032 ceph-mon[24793]: pgmap v37: 97 pgs: 97 active+clean; 339 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 797 op/s 2024-03-24T11:49:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:53 smithi146 ceph-mon[26954]: pgmap v37: 97 pgs: 97 active+clean; 339 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 797 op/s 2024-03-24T11:49:53.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 98186 13.09 MB/sec execute 59 sec latency 33.032 ms 2024-03-24T11:49:54.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 98545 13.06 MB/sec execute 60 sec latency 37.186 ms 2024-03-24T11:49:55.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 99000 13.22 MB/sec execute 61 sec latency 49.102 ms 2024-03-24T11:49:55.539 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:55 smithi032 ceph-mon[24793]: pgmap v38: 97 pgs: 97 active+clean; 339 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.2 MiB/s wr, 519 op/s 2024-03-24T11:49:55.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:55 smithi146 ceph-mon[26954]: pgmap v38: 97 pgs: 97 active+clean; 339 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.2 MiB/s wr, 519 op/s 2024-03-24T11:49:56.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 99394 13.21 MB/sec execute 62 sec latency 41.201 ms 2024-03-24T11:49:57.318 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:57 smithi032 ceph-mon[24793]: pgmap v39: 97 pgs: 97 active+clean; 333 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 776 op/s 2024-03-24T11:49:57.438 INFO:teuthology.orchestra.run.smithi032.stdout: 5 99701 13.16 MB/sec execute 63 sec latency 60.115 ms 2024-03-24T11:49:57.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:57 smithi146 ceph-mon[26954]: pgmap v39: 97 pgs: 97 active+clean; 333 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 776 op/s 2024-03-24T11:49:58.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 99977 13.00 MB/sec execute 64 sec latency 36.256 ms 2024-03-24T11:49:59.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:49:59 smithi032 ceph-mon[24793]: pgmap v40: 97 pgs: 97 active+clean; 333 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-03-24T11:49:59.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 100324 12.91 MB/sec execute 65 sec latency 47.728 ms 2024-03-24T11:49:59.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:49:59 smithi146 ceph-mon[26954]: pgmap v40: 97 pgs: 97 active+clean; 333 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-03-24T11:50:00.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 100651 12.80 MB/sec execute 66 sec latency 34.299 ms 2024-03-24T11:50:00.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:00 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:00.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:00 smithi032 ceph-mon[24793]: overall HEALTH_OK 2024-03-24T11:50:00.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:00 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:00.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:00 smithi146 ceph-mon[26954]: overall HEALTH_OK 2024-03-24T11:50:01.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 101023 12.72 MB/sec execute 67 sec latency 53.379 ms 2024-03-24T11:50:01.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:01 smithi032 ceph-mon[24793]: pgmap v41: 97 pgs: 97 active+clean; 333 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-03-24T11:50:01.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:01 smithi146 ceph-mon[26954]: pgmap v41: 97 pgs: 97 active+clean; 333 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-03-24T11:50:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 101456 12.68 MB/sec execute 68 sec latency 47.503 ms 2024-03-24T11:50:03.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:03 smithi032 ceph-mon[24793]: pgmap v42: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 708 op/s 2024-03-24T11:50:03.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:03 smithi146 ceph-mon[26954]: pgmap v42: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 708 op/s 2024-03-24T11:50:03.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 101933 12.68 MB/sec execute 69 sec latency 56.949 ms 2024-03-24T11:50:04.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 102335 12.71 MB/sec execute 70 sec latency 35.477 ms 2024-03-24T11:50:05.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:05 smithi032 ceph-mon[24793]: pgmap v43: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-03-24T11:50:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 102806 12.80 MB/sec execute 71 sec latency 49.268 ms 2024-03-24T11:50:05.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:05 smithi146 ceph-mon[26954]: pgmap v43: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-03-24T11:50:06.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 103177 12.82 MB/sec execute 72 sec latency 71.360 ms 2024-03-24T11:50:07.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:07 smithi032 ceph-mon[24793]: pgmap v44: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.3 MiB/s wr, 718 op/s 2024-03-24T11:50:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 103491 12.71 MB/sec execute 73 sec latency 42.951 ms 2024-03-24T11:50:07.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:07 smithi146 ceph-mon[26954]: pgmap v44: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.3 MiB/s wr, 718 op/s 2024-03-24T11:50:08.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 103850 12.60 MB/sec execute 74 sec latency 29.637 ms 2024-03-24T11:50:09.384 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:09 smithi032 ceph-mon[24793]: pgmap v45: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 461 op/s 2024-03-24T11:50:09.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 104221 12.55 MB/sec execute 75 sec latency 62.981 ms 2024-03-24T11:50:09.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:09 smithi146 ceph-mon[26954]: pgmap v45: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 461 op/s 2024-03-24T11:50:09.852 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:50:10.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 104607 12.51 MB/sec execute 76 sec latency 41.641 ms 2024-03-24T11:50:10.908 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 starting - - - - 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 71s ago 9m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (9m) 71s ago 9m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 starting - - - - 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (3m) 71s ago 3m 43.4M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (3m) 71s ago 3m 15.0M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:9283 running (10m) 71s ago 10m 386M - 17.2.0 e1d6a67b021e 7b699f0e61e8 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (97s) 71s ago 9m 492M - 19.0.0-1667-gdb0330b1 4f325f8440e9 eced178a5b22 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 71s ago 11m 51.4M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (8m) 71s ago 8m 41.1M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (3m) 71s ago 3m 243M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (3m) 71s ago 3m 78.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 starting - - - - 2024-03-24T11:50:10.909 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (8m) 71s ago 8m 16.0M - 1.3.1 1dbe0e931976 9c0cab415cfb 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 71s ago 8m 287M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 71s ago 7m 222M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 71s ago 7m 166M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (6m) 71s ago 6m 238M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (6m) 71s ago 6m 265M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (6m) 71s ago 6m 289M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (5m) 71s ago 5m 262M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (5m) 71s ago 5m 367M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:50:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8m) 71s ago 8m 64.6M - 2.33.4 514e6a882f6e 5c09869266dc 2024-03-24T11:50:11.402 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:11 smithi032 ceph-mon[24793]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:11.402 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:11 smithi032 ceph-mon[24793]: pgmap v46: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 461 op/s 2024-03-24T11:50:11.402 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:11 smithi032 ceph-mon[24793]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 104906 12.43 MB/sec execute 77 sec latency 49.280 ms 2024-03-24T11:50:11.524 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:50:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:50:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:50:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:50:11.525 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:50:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:11 smithi146 ceph-mon[26954]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:11 smithi146 ceph-mon[26954]: pgmap v46: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 461 op/s 2024-03-24T11:50:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:11 smithi146 ceph-mon[26954]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:12.065 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:50:12.066 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:50:12.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:12 smithi032 ceph-mon[24793]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:12.407 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:12 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1073770415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 105345 12.38 MB/sec execute 78 sec latency 52.932 ms 2024-03-24T11:50:12.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:12 smithi146 ceph-mon[26954]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:12.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:12 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1073770415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:50:13.414 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:13 smithi032 ceph-mon[24793]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:13.414 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:13 smithi032 ceph-mon[24793]: pgmap v47: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-03-24T11:50:13.414 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:13 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1390472104' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:50:13.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 105864 12.47 MB/sec execute 79 sec latency 38.482 ms 2024-03-24T11:50:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:13 smithi146 ceph-mon[26954]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:13.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:13 smithi146 ceph-mon[26954]: pgmap v47: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-03-24T11:50:13.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:13 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1390472104' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:50:14.419 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:14 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:14.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 106364 12.57 MB/sec execute 80 sec latency 91.790 ms 2024-03-24T11:50:14.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:14 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:15.427 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:15 smithi032 ceph-mon[24793]: pgmap v48: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 480 op/s 2024-03-24T11:50:15.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 106862 12.60 MB/sec execute 81 sec latency 34.251 ms 2024-03-24T11:50:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:15 smithi146 ceph-mon[26954]: pgmap v48: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 480 op/s 2024-03-24T11:50:16.432 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:16 smithi032 ceph-mon[24793]: mgrmap e27: smithi146.tfohfl(active, since 92s), standbys: smithi032.xjdgbw 2024-03-24T11:50:16.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 107155 12.50 MB/sec execute 82 sec latency 38.477 ms 2024-03-24T11:50:16.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:16 smithi146 ceph-mon[26954]: mgrmap e27: smithi146.tfohfl(active, since 92s), standbys: smithi032.xjdgbw 2024-03-24T11:50:17.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 107606 12.46 MB/sec execute 83 sec latency 55.369 ms 2024-03-24T11:50:17.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:17 smithi032 ceph-mon[24793]: pgmap v49: 97 pgs: 97 active+clean; 341 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 7.7 MiB/s wr, 736 op/s 2024-03-24T11:50:17.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:17 smithi146 ceph-mon[26954]: pgmap v49: 97 pgs: 97 active+clean; 341 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 7.7 MiB/s wr, 736 op/s 2024-03-24T11:50:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 108137 12.46 MB/sec execute 84 sec latency 31.872 ms 2024-03-24T11:50:18.835 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:18 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:18.835 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:18 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:18.835 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:18 smithi146 ceph-mon[26954]: Reconfiguring node-exporter.smithi146 (dependencies changed)... 2024-03-24T11:50:18.835 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:18 smithi146 ceph-mon[26954]: Reconfiguring daemon node-exporter.smithi146 on smithi146 2024-03-24T11:50:18.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:18 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:18.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:18 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:18.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:18 smithi032 ceph-mon[24793]: Reconfiguring node-exporter.smithi146 (dependencies changed)... 2024-03-24T11:50:18.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:18 smithi032 ceph-mon[24793]: Reconfiguring daemon node-exporter.smithi146 on smithi146 2024-03-24T11:50:19.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 108566 12.40 MB/sec execute 85 sec latency 69.113 ms 2024-03-24T11:50:19.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:19 smithi032 ceph-mon[24793]: pgmap v50: 97 pgs: 97 active+clean; 341 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 482 op/s 2024-03-24T11:50:19.965 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:19 smithi146 ceph-mon[26954]: pgmap v50: 97 pgs: 97 active+clean; 341 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 482 op/s 2024-03-24T11:50:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 109068 12.41 MB/sec execute 86 sec latency 35.372 ms 2024-03-24T11:50:21.023 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.023 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.023 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:50:21.023 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:50:21.023 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:50:21.024 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:50:21.024 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.024 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:50:21.024 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:50:21.024 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:50:21.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:50:21.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:50:21.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:50:21.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:50:21.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:50:21.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:21.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:21.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:21.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:21 smithi032 ceph-mon[24793]: pgmap v51: 97 pgs: 97 active+clean; 341 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 482 op/s 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:50:21.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:21.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:21 smithi146 ceph-mon[26954]: pgmap v51: 97 pgs: 97 active+clean; 341 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 482 op/s 2024-03-24T11:50:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: 5 109344 12.38 MB/sec execute 87 sec latency 59.689 ms 2024-03-24T11:50:22.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 109696 12.44 MB/sec execute 88 sec latency 42.109 ms 2024-03-24T11:50:23.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 110026 12.42 MB/sec execute 89 sec latency 33.481 ms 2024-03-24T11:50:23.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: pgmap v52: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 756 op/s 2024-03-24T11:50:23.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: Upgrade: Updating mgr.smithi032.xjdgbw 2024-03-24T11:50:23.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:23.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:50:23.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:50:23.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:50:23.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:23.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:23 smithi032 ceph-mon[24793]: Deploying daemon mgr.smithi032.xjdgbw on smithi032 2024-03-24T11:50:24.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: pgmap v52: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 756 op/s 2024-03-24T11:50:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: Upgrade: Updating mgr.smithi032.xjdgbw 2024-03-24T11:50:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:50:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:50:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:50:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:24.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:23 smithi146 ceph-mon[26954]: Deploying daemon mgr.smithi032.xjdgbw on smithi032 2024-03-24T11:50:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 110330 12.39 MB/sec execute 90 sec latency 66.704 ms 2024-03-24T11:50:25.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 110579 12.29 MB/sec execute 91 sec latency 47.447 ms 2024-03-24T11:50:25.834 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:25 smithi032 ceph-mon[24793]: pgmap v53: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 529 op/s 2024-03-24T11:50:26.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:25 smithi146 ceph-mon[26954]: pgmap v53: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 529 op/s 2024-03-24T11:50:26.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 110861 12.20 MB/sec execute 92 sec latency 97.492 ms 2024-03-24T11:50:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 111202 12.15 MB/sec execute 93 sec latency 104.255 ms 2024-03-24T11:50:27.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:27 smithi032 ceph-mon[24793]: pgmap v54: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-03-24T11:50:28.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:27 smithi146 ceph-mon[26954]: pgmap v54: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-03-24T11:50:28.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 111650 12.12 MB/sec execute 94 sec latency 74.808 ms 2024-03-24T11:50:29.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 112015 12.09 MB/sec execute 95 sec latency 43.132 ms 2024-03-24T11:50:29.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:29 smithi032 ceph-mon[24793]: pgmap v55: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 463 op/s 2024-03-24T11:50:29.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:29 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:30.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:29 smithi146 ceph-mon[26954]: pgmap v55: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 463 op/s 2024-03-24T11:50:30.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:29 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:30.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 112527 12.10 MB/sec execute 96 sec latency 42.837 ms 2024-03-24T11:50:31.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 112811 12.05 MB/sec execute 97 sec latency 47.210 ms 2024-03-24T11:50:31.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:31 smithi032 ceph-mon[24793]: pgmap v56: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 463 op/s 2024-03-24T11:50:31.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:31 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:31.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:31 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:31.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:31 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:31.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:31 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:31.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:31 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:31.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:31 smithi146 ceph-mon[26954]: pgmap v56: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 463 op/s 2024-03-24T11:50:31.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:31 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:31.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:31 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:31.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:31 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:31.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:31 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:31.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:31 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:32.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 113158 12.10 MB/sec execute 98 sec latency 35.203 ms 2024-03-24T11:50:33.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:33 smithi032 ceph-mon[24793]: pgmap v57: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 700 op/s 2024-03-24T11:50:33.364 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:33 smithi146 ceph-mon[26954]: pgmap v57: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 700 op/s 2024-03-24T11:50:33.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 113478 12.09 MB/sec execute 99 sec latency 39.002 ms 2024-03-24T11:50:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 113801 12.07 MB/sec execute 100 sec latency 71.089 ms 2024-03-24T11:50:35.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 114032 12.01 MB/sec execute 101 sec latency 53.863 ms 2024-03-24T11:50:35.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:35 smithi032 ceph-mon[24793]: pgmap v58: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 426 op/s 2024-03-24T11:50:35.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:35 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:35.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:35 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:35.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:35 smithi146 ceph-mon[26954]: pgmap v58: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 426 op/s 2024-03-24T11:50:35.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:35 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:35.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:35 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:36.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 114390 11.94 MB/sec execute 102 sec latency 36.709 ms 2024-03-24T11:50:37.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 114962 11.93 MB/sec execute 103 sec latency 52.463 ms 2024-03-24T11:50:37.497 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:37 smithi032 ceph-mon[24793]: pgmap v59: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.5 MiB/s wr, 610 op/s 2024-03-24T11:50:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:37 smithi146 ceph-mon[26954]: pgmap v59: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.5 MiB/s wr, 610 op/s 2024-03-24T11:50:38.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 115504 11.96 MB/sec execute 104 sec latency 39.663 ms 2024-03-24T11:50:39.304 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:39 smithi032 ceph-mon[24793]: pgmap v60: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-03-24T11:50:39.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 115954 11.95 MB/sec execute 105 sec latency 69.697 ms 2024-03-24T11:50:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:39 smithi146 ceph-mon[26954]: pgmap v60: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-03-24T11:50:40.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 116364 11.93 MB/sec execute 106 sec latency 39.435 ms 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: pgmap v61: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 455 op/s 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: Standby manager daemon smithi032.xjdgbw restarted 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: Standby manager daemon smithi032.xjdgbw started 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/crt"}]: dispatch 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/key"}]: dispatch 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:41.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:41 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:41.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 116871 12.04 MB/sec execute 107 sec latency 44.534 ms 2024-03-24T11:50:41.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: pgmap v61: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 455 op/s 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: Standby manager daemon smithi032.xjdgbw restarted 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: Standby manager daemon smithi032.xjdgbw started 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/crt"}]: dispatch 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.xjdgbw/key"}]: dispatch 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:41 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:42.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:42 smithi032 ceph-mon[24793]: mgrmap e28: smithi146.tfohfl(active, since 118s), standbys: smithi032.xjdgbw 2024-03-24T11:50:42.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 117274 12.05 MB/sec execute 108 sec latency 43.615 ms 2024-03-24T11:50:42.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:42 smithi146 ceph-mon[26954]: mgrmap e28: smithi146.tfohfl(active, since 118s), standbys: smithi032.xjdgbw 2024-03-24T11:50:43.212 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:50:43.368 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:43 smithi032 ceph-mon[24793]: pgmap v62: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 707 op/s 2024-03-24T11:50:43.369 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:43 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:43.369 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:43 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:43.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 117566 12.01 MB/sec execute 109 sec latency 40.986 ms 2024-03-24T11:50:43.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:43 smithi146 ceph-mon[26954]: pgmap v62: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 707 op/s 2024-03-24T11:50:43.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:43 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:43.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:43 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:44.154 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (82s) 2s ago 10m 24.3M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 2s ago 10m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (9m) 9s ago 9m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (58s) 2s ago 9m 45.5M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (4m) 2s ago 4m 39.3M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (4m) 9s ago 4m 15.5M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (14s) 2s ago 11m 91.9M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (2m) 9s ago 9m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 eced178a5b22 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 2s ago 11m 52.5M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (9m) 9s ago 9m 42.4M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (4m) 2s ago 4m 250M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (3m) 9s ago 3m 84.7M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (55s) 2s ago 9m 8845k - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (24s) 9s ago 9m 9819k - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:50:44.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 2s ago 8m 443M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 2s ago 8m 340M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 2s ago 7m 265M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (7m) 2s ago 7m 401M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (7m) 9s ago 7m 394M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (6m) 9s ago 6m 428M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (6m) 9s ago 6m 369M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (5m) 9s ago 5m 531M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:50:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (27s) 2s ago 9m 54.9M - 2.33.4 514e6a882f6e 84d2cb344c7d 2024-03-24T11:50:44.376 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:44 smithi032 ceph-mon[24793]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:44.377 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:44 smithi032 ceph-mon[24793]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 117916 11.94 MB/sec execute 110 sec latency 41.804 ms 2024-03-24T11:50:44.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:44 smithi146 ceph-mon[26954]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:44.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:44 smithi146 ceph-mon[26954]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:44.844 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:50:44.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:50:44.844 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:50:44.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:44.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:50:44.845 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:50:45.362 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:50:45.362 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:50:45.362 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:50:45.362 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:50:45.362 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:50:45.362 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-03-24T11:50:45.362 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:50:45.363 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T11:50:45.363 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mgr daemons", 2024-03-24T11:50:45.363 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:50:45.363 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:50:45.388 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:45 smithi032 ceph-mon[24793]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:45.388 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:45 smithi032 ceph-mon[24793]: pgmap v63: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-03-24T11:50:45.388 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:45 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:45.389 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:45 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/4059353196' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 118409 11.92 MB/sec execute 111 sec latency 36.933 ms 2024-03-24T11:50:45.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:45 smithi146 ceph-mon[26954]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:45.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:45 smithi146 ceph-mon[26954]: pgmap v63: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-03-24T11:50:45.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:45 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:45.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:45 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/4059353196' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:45.977 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:50:46.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:46 smithi032 ceph-mon[24793]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:46.410 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:46 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/1488131695' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:50:46.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 118961 11.95 MB/sec execute 112 sec latency 33.975 ms 2024-03-24T11:50:46.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:46 smithi146 ceph-mon[26954]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:46.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:46 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1488131695' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:50:47.418 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:47 smithi032 ceph-mon[24793]: pgmap v64: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-03-24T11:50:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 119581 11.96 MB/sec execute 113 sec latency 22.173 ms 2024-03-24T11:50:47.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:47 smithi146 ceph-mon[26954]: pgmap v64: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-03-24T11:50:48.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 120105 12.00 MB/sec execute 114 sec latency 44.779 ms 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: pgmap v65: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr fail", "who": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr fail", "who": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:50:49.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:49 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:50:49.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 120572 12.07 MB/sec execute 115 sec latency 32.383 ms 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: pgmap v65: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 172.21.15.146:0/2118835741' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr fail", "who": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "mgr fail", "who": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:50:49.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:49 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:50:50.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 120899 12.07 MB/sec execute 116 sec latency 46.408 ms 2024-03-24T11:50:50.502 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:50.502 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: Failing over to other MGR 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd='[{"prefix": "mgr fail", "who": "smithi146.tfohfl"}]': finished 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: mgrmap e29: smithi032.xjdgbw(active, starting, since 0.861925s), standbys: smithi146.tfohfl 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi146.tfohfl", "id": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:50.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:50:50.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:50 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: Upgrade: Need to upgrade myself (mgr.smithi146.tfohfl) 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: Failing over to other MGR 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.24437 ' entity='mgr.smithi146.tfohfl' cmd='[{"prefix": "mgr fail", "who": "smithi146.tfohfl"}]': finished 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: mgrmap e29: smithi032.xjdgbw(active, starting, since 0.861925s), standbys: smithi146.tfohfl 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:50:50.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi146.tfohfl", "id": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:50:50.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:50:50.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:50.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:50.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:50.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:50:50.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:50 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:50:51.440 INFO:teuthology.orchestra.run.smithi032.stdout: 5 121097 12.00 MB/sec execute 117 sec latency 95.185 ms 2024-03-24T11:50:52.066 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:51 smithi032 ceph-mon[24793]: mgrmap e30: smithi032.xjdgbw(active, since 1.86736s), standbys: smithi146.tfohfl 2024-03-24T11:50:52.066 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:51 smithi032 ceph-mon[24793]: pgmap v3: 97 pgs: 97 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:50:52.122 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:51 smithi146 ceph-mon[26954]: mgrmap e30: smithi032.xjdgbw(active, since 1.86736s), standbys: smithi146.tfohfl 2024-03-24T11:50:52.122 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:51 smithi146 ceph-mon[26954]: pgmap v3: 97 pgs: 97 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:50:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 121409 11.93 MB/sec execute 118 sec latency 40.651 ms 2024-03-24T11:50:53.331 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:53 smithi032 ceph-mon[24793]: pgmap v4: 97 pgs: 97 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:50:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:53 smithi146 ceph-mon[26954]: pgmap v4: 97 pgs: 97 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:50:53.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 121872 11.92 MB/sec execute 119 sec latency 39.066 ms 2024-03-24T11:50:54.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:54 smithi032 ceph-mon[24793]: mgrmap e31: smithi032.xjdgbw(active, since 4s), standbys: smithi146.tfohfl 2024-03-24T11:50:54.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:54 smithi146 ceph-mon[26954]: mgrmap e31: smithi032.xjdgbw(active, since 4s), standbys: smithi146.tfohfl 2024-03-24T11:50:54.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 122249 11.88 MB/sec execute 120 sec latency 52.856 ms 2024-03-24T11:50:55.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:55 smithi032 ceph-mon[24793]: [24/Mar/2024:11:50:54] ENGINE Bus STARTING 2024-03-24T11:50:55.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:55 smithi032 ceph-mon[24793]: [24/Mar/2024:11:50:54] ENGINE Serving on http://172.21.15.32:8765 2024-03-24T11:50:55.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:55 smithi032 ceph-mon[24793]: [24/Mar/2024:11:50:54] ENGINE Serving on https://172.21.15.32:7150 2024-03-24T11:50:55.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:55 smithi032 ceph-mon[24793]: [24/Mar/2024:11:50:54] ENGINE Bus STARTED 2024-03-24T11:50:55.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:55 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:55.108 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:55 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:55.325 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:55 smithi146 ceph-mon[26954]: [24/Mar/2024:11:50:54] ENGINE Bus STARTING 2024-03-24T11:50:55.325 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:55 smithi146 ceph-mon[26954]: [24/Mar/2024:11:50:54] ENGINE Serving on http://172.21.15.32:8765 2024-03-24T11:50:55.325 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:55 smithi146 ceph-mon[26954]: [24/Mar/2024:11:50:54] ENGINE Serving on https://172.21.15.32:7150 2024-03-24T11:50:55.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:55 smithi146 ceph-mon[26954]: [24/Mar/2024:11:50:54] ENGINE Bus STARTED 2024-03-24T11:50:55.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:55 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:55.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:55 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:55.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 122607 11.86 MB/sec execute 121 sec latency 94.044 ms 2024-03-24T11:50:56.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 122988 11.82 MB/sec execute 122 sec latency 63.189 ms 2024-03-24T11:50:57.174 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:57 smithi032 ceph-mon[24793]: pgmap v6: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 217 KiB/s wr, 16 op/s 2024-03-24T11:50:57.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:57 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:57.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:57 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:57.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:57 smithi146 ceph-mon[26954]: pgmap v6: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 217 KiB/s wr, 16 op/s 2024-03-24T11:50:57.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:57 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:57.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:57 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:57.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 123329 11.81 MB/sec execute 123 sec latency 35.472 ms 2024-03-24T11:50:58.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 123850 11.91 MB/sec execute 124 sec latency 26.655 ms 2024-03-24T11:50:59.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 124298 11.93 MB/sec execute 125 sec latency 31.891 ms 2024-03-24T11:50:59.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: pgmap v7: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 3.2 MiB/s wr, 420 op/s 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:50:59.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:50:59 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:50:59.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: pgmap v7: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 3.2 MiB/s wr, 420 op/s 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:50:59.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:50:59 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3854546270' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:51:00.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 124638 11.90 MB/sec execute 126 sec latency 39.155 ms 2024-03-24T11:51:00.488 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:00 smithi032 ceph-mon[24793]: mgrmap e32: smithi032.xjdgbw(active, since 10s), standbys: smithi146.tfohfl 2024-03-24T11:51:00.488 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:00 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:00.488 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:00 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:00.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:00 smithi146 ceph-mon[26954]: mgrmap e32: smithi032.xjdgbw(active, since 10s), standbys: smithi146.tfohfl 2024-03-24T11:51:00.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:00 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:00.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:00 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:01.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 124879 11.83 MB/sec execute 127 sec latency 92.838 ms 2024-03-24T11:51:01.494 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:01 smithi032 ceph-mon[24793]: pgmap v8: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 2.5 MiB/s wr, 326 op/s 2024-03-24T11:51:01.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:01 smithi146 ceph-mon[26954]: pgmap v8: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 2.5 MiB/s wr, 326 op/s 2024-03-24T11:51:02.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 125181 11.78 MB/sec execute 128 sec latency 44.092 ms 2024-03-24T11:51:03.441 INFO:teuthology.orchestra.run.smithi032.stdout: 5 125486 11.74 MB/sec execute 129 sec latency 49.076 ms 2024-03-24T11:51:03.511 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:03 smithi032 ceph-mon[24793]: pgmap v9: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 521 op/s 2024-03-24T11:51:03.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:03 smithi146 ceph-mon[26954]: pgmap v9: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 521 op/s 2024-03-24T11:51:04.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 125841 11.71 MB/sec execute 130 sec latency 62.032 ms 2024-03-24T11:51:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 126176 11.68 MB/sec execute 131 sec latency 54.831 ms 2024-03-24T11:51:05.549 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:05 smithi032 ceph-mon[24793]: pgmap v10: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.7 MiB/s wr, 475 op/s 2024-03-24T11:51:05.549 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:05 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:05.549 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:05 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:05.549 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:05 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:05.550 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:05 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:05.550 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:05 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:05.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:05 smithi146 ceph-mon[26954]: pgmap v10: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.7 MiB/s wr, 475 op/s 2024-03-24T11:51:05.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:05 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:05.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:05 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:05.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:05 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:05.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:05 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:05.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:05 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:06.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 126529 11.64 MB/sec execute 132 sec latency 65.528 ms 2024-03-24T11:51:06.555 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:06 smithi032 ceph-mon[24793]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:51:06.556 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:06 smithi032 ceph-mon[24793]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:51:06.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:06 smithi146 ceph-mon[26954]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:51:06.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:06 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:51:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 126865 11.64 MB/sec execute 133 sec latency 45.105 ms 2024-03-24T11:51:07.581 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: pgmap v11: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 483 op/s 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:07 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:51:07.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: pgmap v11: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 483 op/s 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:07.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:07 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:08.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 127279 11.67 MB/sec execute 134 sec latency 31.062 ms 2024-03-24T11:51:08.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:08 smithi032 ceph-mon[24793]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-03-24T11:51:08.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:08 smithi032 ceph-mon[24793]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-03-24T11:51:08.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:08 smithi146 ceph-mon[26954]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-03-24T11:51:08.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:08 smithi146 ceph-mon[26954]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-03-24T11:51:09.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 127670 11.70 MB/sec execute 135 sec latency 37.437 ms 2024-03-24T11:51:09.638 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:09 smithi032 ceph-mon[24793]: pgmap v12: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.1 MiB/s wr, 671 op/s 2024-03-24T11:51:09.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:09 smithi146 ceph-mon[26954]: pgmap v12: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.1 MiB/s wr, 671 op/s 2024-03-24T11:51:10.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 128025 11.71 MB/sec execute 136 sec latency 50.228 ms 2024-03-24T11:51:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 128379 11.66 MB/sec execute 137 sec latency 71.114 ms 2024-03-24T11:51:11.662 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:11 smithi032 ceph-mon[24793]: pgmap v13: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-03-24T11:51:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:11 smithi146 ceph-mon[26954]: pgmap v13: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-03-24T11:51:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 128746 11.62 MB/sec execute 138 sec latency 47.551 ms 2024-03-24T11:51:13.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 129008 11.58 MB/sec execute 139 sec latency 73.230 ms 2024-03-24T11:51:13.754 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:13 smithi032 ceph-mon[24793]: pgmap v14: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.2 MiB/s wr, 652 op/s 2024-03-24T11:51:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:13 smithi146 ceph-mon[26954]: pgmap v14: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.2 MiB/s wr, 652 op/s 2024-03-24T11:51:14.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 129341 11.54 MB/sec execute 140 sec latency 70.705 ms 2024-03-24T11:51:15.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 129682 11.53 MB/sec execute 141 sec latency 57.690 ms 2024-03-24T11:51:15.584 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:15 smithi032 ceph-mon[24793]: pgmap v15: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 419 op/s 2024-03-24T11:51:15.585 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:15 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:15.585 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:15 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:15.585 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:15 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:51:15.585 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:15 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:15.759 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:15 smithi146 ceph-mon[26954]: pgmap v15: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 419 op/s 2024-03-24T11:51:15.759 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:15 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:15.759 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:15 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:15.759 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:15 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:51:15.760 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:15 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:16.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 129938 11.47 MB/sec execute 142 sec latency 83.836 ms 2024-03-24T11:51:16.458 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:51:16.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:16 smithi032 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:51:16.764 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:16 smithi146 ceph-mon[26954]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:51:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 130344 11.48 MB/sec execute 143 sec latency 34.231 ms 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (115s) 19s ago 10m 24.3M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 19s ago 10m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (10m) 22s ago 10m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (91s) 19s ago 10m 45.7M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (4m) 19s ago 4m 39.9M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (4m) 22s ago 4m 15.8M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (47s) 19s ago 12m 478M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283 running (2m) 22s ago 10m 74.3M - 19.0.0-1667-gdb0330b1 4f325f8440e9 eced178a5b22 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 19s ago 12m 54.0M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (10m) 22s ago 10m 43.7M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (4m) 19s ago 4m 249M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:51:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (4m) 22s ago 4m 86.6M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (88s) 19s ago 10m 8786k - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (57s) 22s ago 9m 11.4M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 19s ago 9m 449M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 19s ago 8m 388M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 19s ago 8m 268M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (7m) 19s ago 7m 413M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (7m) 22s ago 7m 430M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (7m) 22s ago 7m 456M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (6m) 22s ago 6m 409M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (6m) 22s ago 6m 551M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:51:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 starting - - - - 2024-03-24T11:51:17.626 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:17 smithi032 ceph-mon[24793]: pgmap v16: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 427 op/s 2024-03-24T11:51:17.626 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:17 smithi032 ceph-mon[24793]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:17.627 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:17 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:51:17.627 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:17 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:51:17.627 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:17 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:17.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:17 smithi146 ceph-mon[26954]: pgmap v16: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 427 op/s 2024-03-24T11:51:17.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:17 smithi146 ceph-mon[26954]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:17.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:17 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:51:17.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:17 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:51:17.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:17 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:18.039 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:51:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:51:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:51:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:51:18.041 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:51:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 130719 11.50 MB/sec execute 144 sec latency 43.242 ms 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mgr daemons", 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:51:18.519 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:51:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:18 smithi146 ceph-mon[26954]: Upgrade: Updating mgr.smithi146.tfohfl 2024-03-24T11:51:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:18 smithi146 ceph-mon[26954]: Deploying daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:51:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:18 smithi146 ceph-mon[26954]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:18 smithi146 ceph-mon[26954]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:18 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/2182117609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:51:18.640 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:18 smithi032 ceph-mon[24793]: Upgrade: Updating mgr.smithi146.tfohfl 2024-03-24T11:51:18.640 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:18 smithi032 ceph-mon[24793]: Deploying daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:51:18.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:18 smithi032 ceph-mon[24793]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:18.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:18 smithi032 ceph-mon[24793]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:18.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:18 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2182117609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:51:19.043 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:51:19.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 131097 11.53 MB/sec execute 145 sec latency 38.269 ms 2024-03-24T11:51:19.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:19 smithi146 ceph-mon[26954]: pgmap v17: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 621 op/s 2024-03-24T11:51:19.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:19 smithi146 ceph-mon[26954]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:19.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:19 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/2488351279' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:51:19.650 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:19 smithi032 ceph-mon[24793]: pgmap v17: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 621 op/s 2024-03-24T11:51:19.650 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:19 smithi032 ceph-mon[24793]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:19.651 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:19 smithi032 ceph-mon[24793]: from='client.? 172.21.15.32:0/2488351279' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:51:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 131522 11.56 MB/sec execute 146 sec latency 63.595 ms 2024-03-24T11:51:20.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:20 smithi146 ceph-mon[26954]: pgmap v18: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-03-24T11:51:20.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:20 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:20.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:20 smithi032 ceph-mon[24793]: pgmap v18: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-03-24T11:51:20.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:20 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 131806 11.51 MB/sec execute 147 sec latency 46.367 ms 2024-03-24T11:51:22.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 132050 11.45 MB/sec execute 148 sec latency 37.760 ms 2024-03-24T11:51:22.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:22 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:22.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:22 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:22.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:22 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:22.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:22 smithi032 ceph-mon[24793]: pgmap v19: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.1 MiB/s wr, 640 op/s 2024-03-24T11:51:22.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:22 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:22.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:22 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:22.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:22 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:22.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:22 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:22.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:22 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:22.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:22 smithi146 ceph-mon[26954]: pgmap v19: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.1 MiB/s wr, 640 op/s 2024-03-24T11:51:22.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:22 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:22.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:22 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:23.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 132244 11.39 MB/sec execute 149 sec latency 89.048 ms 2024-03-24T11:51:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 132465 11.37 MB/sec execute 150 sec latency 100.045 ms 2024-03-24T11:51:24.989 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:24 smithi146 ceph-mon[26954]: pgmap v20: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.3 MiB/s wr, 422 op/s 2024-03-24T11:51:25.219 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:24 smithi032 ceph-mon[24793]: pgmap v20: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.3 MiB/s wr, 422 op/s 2024-03-24T11:51:25.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 132679 11.32 MB/sec execute 151 sec latency 81.270 ms 2024-03-24T11:51:26.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 132924 11.26 MB/sec execute 152 sec latency 91.078 ms 2024-03-24T11:51:27.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:26 smithi032 ceph-mon[24793]: pgmap v21: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-03-24T11:51:27.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:26 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:27.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:26 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:27.101 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:26 smithi146 ceph-mon[26954]: pgmap v21: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-03-24T11:51:27.101 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:26 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:27.101 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:26 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 133210 11.24 MB/sec execute 153 sec latency 67.342 ms 2024-03-24T11:51:28.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 133519 11.20 MB/sec execute 154 sec latency 71.551 ms 2024-03-24T11:51:29.292 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:29 smithi146 ceph-mon[26954]: pgmap v22: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 576 op/s 2024-03-24T11:51:29.292 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:29 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:29.293 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:29 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:29.381 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:29 smithi032 ceph-mon[24793]: pgmap v22: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 576 op/s 2024-03-24T11:51:29.381 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:29 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:29.381 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:29 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:29.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 133958 11.23 MB/sec execute 155 sec latency 36.881 ms 2024-03-24T11:51:30.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 134326 11.23 MB/sec execute 156 sec latency 41.875 ms 2024-03-24T11:51:30.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: pgmap v23: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 374 op/s 2024-03-24T11:51:30.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:30.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:30.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:51:30.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:51:30.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:51:30.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:51:30.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:51:30.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:30 smithi032 ceph-mon[24793]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:51:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: pgmap v23: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 374 op/s 2024-03-24T11:51:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:51:31.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:51:31.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:51:31.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:51:31.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:51:31.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:30 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/800499143' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:51:31.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 134722 11.28 MB/sec execute 157 sec latency 43.806 ms 2024-03-24T11:51:32.293 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:32 smithi146 ceph-mon[26954]: mgrmap e33: smithi032.xjdgbw(active, since 42s), standbys: smithi146.tfohfl 2024-03-24T11:51:32.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 135078 11.29 MB/sec execute 158 sec latency 45.075 ms 2024-03-24T11:51:32.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:32 smithi032 ceph-mon[24793]: mgrmap e33: smithi032.xjdgbw(active, since 42s), standbys: smithi146.tfohfl 2024-03-24T11:51:33.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 135331 11.24 MB/sec execute 159 sec latency 41.624 ms 2024-03-24T11:51:33.491 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: pgmap v24: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.2 MiB/s wr, 573 op/s 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.xjdgbw"}]: dispatch 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.xjdgbw"}]': finished 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi146.tfohfl"}]: dispatch 2024-03-24T11:51:33.492 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi146.tfohfl"}]': finished 2024-03-24T11:51:33.493 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:33 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:51:33.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: pgmap v24: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.2 MiB/s wr, 573 op/s 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.xjdgbw"}]: dispatch 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.xjdgbw"}]': finished 2024-03-24T11:51:33.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi146.tfohfl"}]: dispatch 2024-03-24T11:51:33.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi146.tfohfl"}]': finished 2024-03-24T11:51:33.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:33 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:51:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 135612 11.19 MB/sec execute 160 sec latency 35.857 ms 2024-03-24T11:51:34.517 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:34 smithi032 ceph-mon[24793]: Upgrade: Setting container_image for all mgr 2024-03-24T11:51:34.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:34 smithi146 ceph-mon[26954]: Upgrade: Setting container_image for all mgr 2024-03-24T11:51:35.442 INFO:teuthology.orchestra.run.smithi032.stdout: 5 135926 11.16 MB/sec execute 161 sec latency 42.405 ms 2024-03-24T11:51:35.522 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:35 smithi032 ceph-mon[24793]: pgmap v25: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.4 MiB/s wr, 360 op/s 2024-03-24T11:51:35.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:35 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:35.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:35 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:51:35.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:35 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:51:35.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:35 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:35.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:35 smithi032 ceph-mon[24793]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:35 smithi146 ceph-mon[26954]: pgmap v25: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.4 MiB/s wr, 360 op/s 2024-03-24T11:51:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:35 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:51:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:35 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:51:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:35 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:51:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:35 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:35 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:36.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 136269 11.15 MB/sec execute 162 sec latency 53.856 ms 2024-03-24T11:51:36.532 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:36 smithi032 ceph-mon[24793]: Upgrade: Updating mon.smithi032 2024-03-24T11:51:36.532 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:36 smithi032 ceph-mon[24793]: Deploying daemon mon.smithi032 on smithi032 2024-03-24T11:51:36.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:36 smithi146 ceph-mon[26954]: Upgrade: Updating mon.smithi032 2024-03-24T11:51:36.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:36 smithi146 ceph-mon[26954]: Deploying daemon mon.smithi032 on smithi032 2024-03-24T11:51:37.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 136778 11.14 MB/sec execute 163 sec latency 44.865 ms 2024-03-24T11:51:37.537 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:37 smithi032 ceph-mon[24793]: pgmap v26: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 422 op/s 2024-03-24T11:51:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:37 smithi146 ceph-mon[26954]: pgmap v26: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 422 op/s 2024-03-24T11:51:38.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 137399 11.16 MB/sec execute 164 sec latency 36.033 ms 2024-03-24T11:51:38.806 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:38 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:39 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[24789]: 2024-03-24T11:51:39.199+0000 7f71f8e16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:51:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:39 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[24789]: 2024-03-24T11:51:39.199+0000 7f71f8e16700 -1 mon.smithi032@0(leader) e2 *** Got Signal Terminated *** 2024-03-24T11:51:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:39 smithi032 podman[66289]: 2024-03-24 11:51:39.242412476 +0000 UTC m=+0.226755910 container died 4da956fd8f670ef3a51012f8e9b434728a4c6a98172da7a7084ee96c2d500031 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, ceph=True, version=8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, io.openshift.expose-services=) 2024-03-24T11:51:39.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 137900 11.20 MB/sec execute 165 sec latency 31.997 ms 2024-03-24T11:51:40.319 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:40 smithi032 podman[66289]: 2024-03-24 11:51:40.111694273 +0000 UTC m=+1.096037670 container cleanup 4da956fd8f670ef3a51012f8e9b434728a4c6a98172da7a7084ee96c2d500031 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.expose-services=, RELEASE=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-24T11:51:40.320 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:40 smithi032 bash[66289]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032 2024-03-24T11:51:40.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 138448 11.28 MB/sec execute 166 sec latency 34.217 ms 2024-03-24T11:51:41.078 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:40 smithi032 podman[66304]: 2024-03-24 11:51:40.829401987 +0000 UTC m=+1.580372253 container remove 4da956fd8f670ef3a51012f8e9b434728a4c6a98172da7a7084ee96c2d500031 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, RELEASE=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, version=8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T11:51:41.079 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:41 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service: Deactivated successfully. 2024-03-24T11:51:41.079 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:41 smithi032 systemd[1]: Stopped Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:51:41.079 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:41 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service: Consumed 16.378s CPU time. 2024-03-24T11:51:41.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 138913 11.27 MB/sec execute 167 sec latency 43.723 ms 2024-03-24T11:51:41.581 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:41 smithi032 systemd[1]: Starting Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:41.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:41 smithi032 podman[66437]: 2024-03-24 11:51:41.527674638 +0000 UTC m=+0.031728061 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:51:41.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:41 smithi032 podman[66437]: 2024-03-24 11:51:41.707359399 +0000 UTC m=+0.211412802 container create c8989f11d4c04750b94923e5a91fc6d640e853710c9d6ac377288a87077494d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:51:42.346 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 podman[66437]: 2024-03-24 11:51:42.325218345 +0000 UTC m=+0.829271754 container init c8989f11d4c04750b94923e5a91fc6d640e853710c9d6ac377288a87077494d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T11:51:42.346 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 podman[66437]: 2024-03-24 11:51:42.339245896 +0000 UTC m=+0.843299308 container start c8989f11d4c04750b94923e5a91fc6d640e853710c9d6ac377288a87077494d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:51:42.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 139369 11.26 MB/sec execute 168 sec latency 53.888 ms 2024-03-24T11:51:42.599 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:51:42.599 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T11:51:42.599 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: pidfile_write: ignore empty --pid-file 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: load: jerasure load: lrc 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Git sha 0 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: DB SUMMARY 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: DB Session ID: 55PAD6C66KQPJEOLNMV2 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi032/store.db dir, Total Num: 1, files: 000030.sst 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi032/store.db: 000028.log size: 2480668 ; 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:51:42.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.env: 0x564c325f1ae0 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.info_log: 0x564c34895760 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.statistics: (nil) 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.use_fsync: 0 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:51:42.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.db_log_dir: 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.wal_dir: 2024-03-24T11:51:42.602 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:51:42.603 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:51:42.603 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.write_buffer_manager: 0x564c34869400 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.unordered_write: 0 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:51:42.604 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.row_cache: None 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.wal_filter: None 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.wal_compression: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:51:42.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:51:42.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_open_files: -1 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Compression algorithms supported: 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kZSTD supported: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:51:42.607 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000015 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.merge_operator: 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_filter: None 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564c34895860) 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_type: 0 2024-03-24T11:51:42.608 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_index_type: 0 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_shortening: 1 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: checksum: 4 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: no_block_cache: 0 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache: 0x564c34899090 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_options: 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: capacity : 536870912 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_shard_bits : 4 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: strict_capacity_limit : 0 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_compressed: (nil) 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: persistent_cache: (nil) 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size: 4096 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size_deviation: 10 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_restart_interval: 16 2024-03-24T11:51:42.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_block_restart_interval: 1 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_block_size: 4096 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: partition_filters: 0 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: use_delta_encoding: 1 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: filter_policy: bloomfilter 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: whole_key_filtering: 1 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: verify_compression: 0 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: format_version: 5 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_index_compression: 1 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_align: 0 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_auto_readahead_size: 262144 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: prepopulate_block_cache: 0 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:51:42.610 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression: NoCompression 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.num_levels: 7 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:51:42.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:51:42.612 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:51:42.613 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:51:42.614 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.ttl: 2592000 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:51:42.615 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11835, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fca7b695-5727-4e2a-9c91-a2f9ba2d792a 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711281102398580, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-24T11:51:42.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711281102411006, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2280884, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11836, "largest_seqno": 12856, "table_properties": {"data_size": 2275636, "index_size": 3077, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1157, "raw_key_size": 10158, "raw_average_key_size": 23, "raw_value_size": 2265976, "raw_average_value_size": 5161, "num_data_blocks": 141, "num_entries": 439, "num_filter_entries": 439, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711281102, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fca7b695-5727-4e2a-9c91-a2f9ba2d792a", "db_session_id": "55PAD6C66KQPJEOLNMV2", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711281102411200, "job": 1, "event": "recovery_finished"} 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi032/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564c34988000 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: DB pointer 0x564c34974000 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** DB Stats ** 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:51:42.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 1/0 2.18 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 297.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L6 1/0 6.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 8.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 297.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 297.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 297.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.618 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 105.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 105.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Block cache BinnedLRUCache@0x564c34899090#2 capacity: 512.00 MB usage: 4.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,3.33 KB,0.000634789%) Misc(1,0.00 KB,0%) 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:51:42.619 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: starting mon.smithi032 rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi032 fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???) e2 preinit fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???).mds e7 new map 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???).mds e7 print_map 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: e7 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: legacy client fscid: 1 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Filesystem 'foofs' (1) 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: fs_name foofs 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: epoch 5 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: created 2024-03-24T11:46:17.024254+0000 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: modified 2024-03-24T11:46:22.041887+0000 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: tableserver 0 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: root 0 2024-03-24T11:51:42.620 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: session_timeout 60 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: session_autoclose 300 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_file_size 1099511627776 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_xattr_size 65536 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: required_client_features {} 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: last_failure 0 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: last_failure_osd_epoch 0 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_mds 1 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: in 0 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: up {0=14486} 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: failed 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: damaged 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: stopped 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_pools [3] 2024-03-24T11:51:42.621 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_pool 2 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: inline_data disabled 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: balancer 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: bal_rank_mask -1 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: standby_count_wanted 1 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.foofs.smithi032.qzdcjy{0:14486} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.32:6834/3101395579,v1:172.21.15.32:6835/3101395579] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Standby daemons: 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.foofs.smithi146.xsrsue{-1:24279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???).osd e71 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:51:42.622 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:51:42.623 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 ceph-mon[66455]: mon.smithi032@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-03-24T11:51:42.623 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 bash[66437]: c8989f11d4c04750b94923e5a91fc6d640e853710c9d6ac377288a87077494d5 2024-03-24T11:51:42.623 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:42 smithi032 systemd[1]: Started Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:51:43.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 139913 11.26 MB/sec execute 169 sec latency 30.637 ms 2024-03-24T11:51:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 140186 11.22 MB/sec execute 170 sec latency 941.075 ms 2024-03-24T11:51:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 140186 11.15 MB/sec execute 171 sec latency 1941.110 ms 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: pgmap v28: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 401 op/s 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: pgmap v29: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 7.4 MiB/s wr, 696 op/s 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: pgmap v30: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: mon.smithi032 calling monitor election 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: mon.smithi032 is new leader, mons smithi032,smithi146 in quorum (ranks 0,1) 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi146=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:51:45.712 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: mgrmap e33: smithi032.xjdgbw(active, since 56s), standbys: smithi146.tfohfl 2024-03-24T11:51:45.713 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: overall HEALTH_OK 2024-03-24T11:51:45.713 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: from='mgr.14774 ' entity='' 2024-03-24T11:51:45.713 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:45 smithi032 ceph-mon[66455]: mgrmap e34: smithi032.xjdgbw(active, since 56s), standbys: smithi146.tfohfl 2024-03-24T11:51:45.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: from='mgr.14774 172.21.15.32:0/298319554' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: pgmap v28: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 401 op/s 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: pgmap v29: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 7.4 MiB/s wr, 696 op/s 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: pgmap v30: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: mon.smithi032 calling monitor election 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: mon.smithi032 is new leader, mons smithi032,smithi146 in quorum (ranks 0,1) 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi146=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: mgrmap e33: smithi032.xjdgbw(active, since 56s), standbys: smithi146.tfohfl 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: overall HEALTH_OK 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: from='mgr.14774 ' entity='' 2024-03-24T11:51:45.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:45 smithi146 ceph-mon[26954]: mgrmap e34: smithi032.xjdgbw(active, since 56s), standbys: smithi146.tfohfl 2024-03-24T11:51:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 140572 11.14 MB/sec execute 172 sec latency 1957.145 ms 2024-03-24T11:51:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 141188 11.18 MB/sec execute 173 sec latency 30.655 ms 2024-03-24T11:51:47.735 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:47 smithi032 ceph-mon[66455]: mgrmap e35: smithi032.xjdgbw(active, since 57s), standbys: smithi146.tfohfl 2024-03-24T11:51:47.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:47 smithi146 ceph-mon[26954]: mgrmap e35: smithi032.xjdgbw(active, since 57s), standbys: smithi146.tfohfl 2024-03-24T11:51:48.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 141666 11.24 MB/sec execute 174 sec latency 30.201 ms 2024-03-24T11:51:49.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 142030 11.24 MB/sec execute 175 sec latency 53.872 ms 2024-03-24T11:51:50.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 142446 11.23 MB/sec execute 176 sec latency 63.499 ms 2024-03-24T11:51:51.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 142711 11.21 MB/sec execute 177 sec latency 56.505 ms 2024-03-24T11:51:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: 5 143111 11.20 MB/sec execute 178 sec latency 53.087 ms 2024-03-24T11:51:53.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 143626 11.17 MB/sec execute 179 sec latency 45.899 ms 2024-03-24T11:51:54.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:53 smithi032 ceph-mon[66455]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:51:54.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:53 smithi032 ceph-mon[66455]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:51:54.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:53 smithi032 ceph-mon[66455]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:51:54.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:53 smithi032 ceph-mon[66455]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:51:54.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:53 smithi032 ceph-mon[66455]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:51:54.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:53 smithi032 ceph-mon[66455]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:51:54.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:53 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl restarted 2024-03-24T11:51:54.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:53 smithi146 ceph-mon[26954]: Standby manager daemon smithi146.tfohfl started 2024-03-24T11:51:54.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/crt"}]: dispatch 2024-03-24T11:51:54.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:51:54.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.tfohfl/key"}]: dispatch 2024-03-24T11:51:54.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:53 smithi146 ceph-mon[26954]: from='mgr.? 172.21.15.146:0/3240484415' entity='mgr.smithi146.tfohfl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:51:54.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 144070 11.17 MB/sec execute 180 sec latency 34.070 ms 2024-03-24T11:51:55.097 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:54 smithi032 ceph-mon[66455]: mgrmap e36: smithi032.xjdgbw(active, since 64s), standbys: smithi146.tfohfl 2024-03-24T11:51:55.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:54 smithi146 ceph-mon[26954]: mgrmap e36: smithi032.xjdgbw(active, since 64s), standbys: smithi146.tfohfl 2024-03-24T11:51:55.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 144666 11.21 MB/sec execute 181 sec latency 32.731 ms 2024-03-24T11:51:55.876 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146[26946]: 2024-03-24T11:51:55.582+0000 7fe91f5b1700 -1 mon.smithi146@1(peon).osd e72 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-24T11:51:55.921 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: Active manager daemon smithi032.xjdgbw restarted 2024-03-24T11:51:55.921 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: Activating manager daemon smithi032.xjdgbw 2024-03-24T11:51:55.921 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: mgrmap e37: smithi032.xjdgbw(active, starting, since 0.00580537s), standbys: smithi146.tfohfl 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi146.tfohfl", "id": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:51:55.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: failed to encode map e72 with expected crc 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:51:55.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:51:56.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: Active manager daemon smithi032.xjdgbw restarted 2024-03-24T11:51:56.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: Activating manager daemon smithi032.xjdgbw 2024-03-24T11:51:56.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: mgrmap e37: smithi032.xjdgbw(active, starting, since 0.00580537s), standbys: smithi146.tfohfl 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi032.xjdgbw", "id": "smithi032.xjdgbw"}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr metadata", "who": "smithi146.tfohfl", "id": "smithi146.tfohfl"}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:51:56.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: failed to encode map e72 with expected crc 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: Manager daemon smithi032.xjdgbw is now available 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:51:56.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:55 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.xjdgbw/trash_purge_schedule"}]: dispatch 2024-03-24T11:51:56.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 144872 11.18 MB/sec execute 182 sec latency 68.407 ms 2024-03-24T11:51:56.617 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:51:56.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:56 smithi032 ceph-mon[66455]: failed to encode map e72 with expected crc 2024-03-24T11:51:56.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:56 smithi032 ceph-mon[66455]: failed to encode map e72 with expected crc 2024-03-24T11:51:56.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:56 smithi032 ceph-mon[66455]: failed to encode map e72 with expected crc 2024-03-24T11:51:56.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:56 smithi032 ceph-mon[66455]: [24/Mar/2024:11:51:56] ENGINE Bus STARTING 2024-03-24T11:51:56.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:56 smithi032 ceph-mon[66455]: [24/Mar/2024:11:51:56] ENGINE Serving on http://172.21.15.32:8765 2024-03-24T11:51:56.949 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:56 smithi032 ceph-mon[66455]: mgrmap e38: smithi032.xjdgbw(active, since 1.01181s), standbys: smithi146.tfohfl 2024-03-24T11:51:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:56 smithi146 ceph-mon[26954]: failed to encode map e72 with expected crc 2024-03-24T11:51:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:56 smithi146 ceph-mon[26954]: failed to encode map e72 with expected crc 2024-03-24T11:51:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:56 smithi146 ceph-mon[26954]: failed to encode map e72 with expected crc 2024-03-24T11:51:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:56 smithi146 ceph-mon[26954]: [24/Mar/2024:11:51:56] ENGINE Bus STARTING 2024-03-24T11:51:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:56 smithi146 ceph-mon[26954]: [24/Mar/2024:11:51:56] ENGINE Serving on http://172.21.15.32:8765 2024-03-24T11:51:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:56 smithi146 ceph-mon[26954]: mgrmap e38: smithi032.xjdgbw(active, since 1.01181s), standbys: smithi146.tfohfl 2024-03-24T11:51:57.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 145223 11.21 MB/sec execute 183 sec latency 49.464 ms 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (2m) 27s ago 11m 24.3M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 27s ago 11m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (10m) 30s ago 10m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (2m) 27s ago 11m 44.7M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (5m) 27s ago 5m 40.0M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (5m) 30s ago 5m 16.0M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (87s) 27s ago 12m 494M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (37s) 30s ago 10m 91.3M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 27s ago 12m 54.2M 2048M 17.2.0 e1d6a67b021e 4da956fd8f67 2024-03-24T11:51:57.712 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (10m) 30s ago 10m 43.2M 2048M 17.2.0 e1d6a67b021e d29e2f462ce7 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (5m) 27s ago 5m 249M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (5m) 30s ago 5m 86.6M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (2m) 27s ago 11m 10.9M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (98s) 30s ago 10m 12.7M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 27s ago 10m 482M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (9m) 27s ago 9m 401M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 27s ago 9m 277M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (8m) 27s ago 8m 441M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (8m) 30s ago 8m 488M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (7m) 30s ago 7m 508M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (7m) 30s ago 7m 430M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (7m) 30s ago 7m 342M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:51:57.713 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (43s) 27s ago 10m 49.0M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:51:57.986 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:57 smithi032 ceph-mon[66455]: from='client.34106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:57.987 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:57 smithi032 ceph-mon[66455]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:51:57.987 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:57 smithi032 ceph-mon[66455]: [24/Mar/2024:11:51:56] ENGINE Serving on https://172.21.15.32:7150 2024-03-24T11:51:57.987 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:57 smithi032 ceph-mon[66455]: [24/Mar/2024:11:51:56] ENGINE Bus STARTED 2024-03-24T11:51:57.987 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:57 smithi032 ceph-mon[66455]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:58.019 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:57 smithi146 ceph-mon[26954]: from='client.34106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:58.019 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:57 smithi146 ceph-mon[26954]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:51:58.020 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:57 smithi146 ceph-mon[26954]: [24/Mar/2024:11:51:56] ENGINE Serving on https://172.21.15.32:7150 2024-03-24T11:51:58.020 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:57 smithi146 ceph-mon[26954]: [24/Mar/2024:11:51:56] ENGINE Bus STARTED 2024-03-24T11:51:58.020 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:57 smithi146 ceph-mon[26954]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:58.268 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:51:58.269 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:51:58.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:51:58.269 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:51:58.269 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-24T11:51:58.269 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:51:58.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:51:58.269 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:51:58.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 145510 11.20 MB/sec execute 184 sec latency 41.317 ms 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T11:51:58.727 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-03-24T11:51:58.728 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:51:58.728 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:51:59.017 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:58 smithi032 ceph-mon[66455]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:51:59.018 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:58 smithi032 ceph-mon[66455]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:59.018 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:58 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/1380105843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:51:59.018 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:58 smithi032 ceph-mon[66455]: mgrmap e39: smithi032.xjdgbw(active, since 3s), standbys: smithi146.tfohfl 2024-03-24T11:51:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:58 smithi146 ceph-mon[26954]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:51:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:58 smithi146 ceph-mon[26954]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:51:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:58 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/1380105843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:51:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:58 smithi146 ceph-mon[26954]: mgrmap e39: smithi032.xjdgbw(active, since 3s), standbys: smithi146.tfohfl 2024-03-24T11:51:59.285 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:51:59.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 145808 11.19 MB/sec execute 185 sec latency 50.909 ms 2024-03-24T11:52:00.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:59 smithi032 ceph-mon[66455]: from='client.34142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:00.057 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:51:59 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2501927743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:52:00.114 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:59 smithi146 ceph-mon[26954]: from='client.34142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:00.114 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:51:59 smithi146 ceph-mon[26954]: from='client.? 172.21.15.32:0/2501927743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:52:00.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 146154 11.16 MB/sec execute 186 sec latency 62.416 ms 2024-03-24T11:52:01.071 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:00 smithi032 ceph-mon[66455]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:52:01.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:00 smithi032 ceph-mon[66455]: mgrmap e40: smithi032.xjdgbw(active, since 4s), standbys: smithi146.tfohfl 2024-03-24T11:52:01.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:01.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:00 smithi146 ceph-mon[26954]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:52:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:00 smithi146 ceph-mon[26954]: mgrmap e40: smithi032.xjdgbw(active, since 4s), standbys: smithi146.tfohfl 2024-03-24T11:52:01.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:00 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:01.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:00 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:01.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 146587 11.17 MB/sec execute 187 sec latency 42.606 ms 2024-03-24T11:52:02.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 147062 11.14 MB/sec execute 188 sec latency 49.694 ms 2024-03-24T11:52:02.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:02 smithi032 ceph-mon[66455]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:52:02.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:02.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:02.945 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:02 smithi146 ceph-mon[26954]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-03-24T11:52:02.945 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:02 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:02.945 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:02 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:03.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 147538 11.14 MB/sec execute 189 sec latency 81.040 ms 2024-03-24T11:52:04.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 148060 11.17 MB/sec execute 190 sec latency 33.553 ms 2024-03-24T11:52:04.876 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:04 smithi032 ceph-mon[66455]: pgmap v7: 97 pgs: 97 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 389 op/s 2024-03-24T11:52:04.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:04 smithi146 ceph-mon[26954]: pgmap v7: 97 pgs: 97 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 389 op/s 2024-03-24T11:52:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 148446 11.16 MB/sec execute 191 sec latency 38.438 ms 2024-03-24T11:52:06.444 INFO:teuthology.orchestra.run.smithi032.stdout: 5 148874 11.22 MB/sec execute 192 sec latency 58.161 ms 2024-03-24T11:52:07.166 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:06 smithi032 ceph-mon[66455]: pgmap v8: 97 pgs: 97 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 3.0 MiB/s wr, 303 op/s 2024-03-24T11:52:07.166 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.166 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.166 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.166 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:06 smithi146 ceph-mon[26954]: pgmap v8: 97 pgs: 97 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 3.0 MiB/s wr, 303 op/s 2024-03-24T11:52:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:06 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:06 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:06 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:06 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 149165 11.19 MB/sec execute 193 sec latency 40.271 ms 2024-03-24T11:52:08.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 149507 11.18 MB/sec execute 194 sec latency 42.331 ms 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: pgmap v9: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.5 MiB/s wr, 540 op/s 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:52:08.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:08 smithi032 ceph-mon[66455]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: pgmap v9: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.5 MiB/s wr, 540 op/s 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: Updating smithi032:/etc/ceph/ceph.conf 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.conf 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:52:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:08 smithi146 ceph-mon[26954]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:52:09.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 149709 11.14 MB/sec execute 195 sec latency 53.221 ms 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.227 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:10.228 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:52:10.228 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:10 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:52:10.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:52:10.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: Updating smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:52:10.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: Updating smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:52:10.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:52:10.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:52:10.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 149963 11.10 MB/sec execute 196 sec latency 80.216 ms 2024-03-24T11:52:11.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:11 smithi032 ceph-mon[66455]: pgmap v10: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.1 MiB/s wr, 494 op/s 2024-03-24T11:52:11.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:11.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:11.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:52:11.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:52:11.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:11.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:11 smithi146 ceph-mon[26954]: pgmap v10: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.1 MiB/s wr, 494 op/s 2024-03-24T11:52:11.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:11 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:11.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:11 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:11.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:11 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:52:11.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:11 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:52:11.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:11 smithi146 ceph-mon[26954]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 150711 11.12 MB/sec execute 197 sec latency 23.668 ms 2024-03-24T11:52:12.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:12 smithi032 ceph-mon[66455]: Upgrade: Updating mon.smithi146 2024-03-24T11:52:12.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:12 smithi032 ceph-mon[66455]: Deploying daemon mon.smithi146 on smithi146 2024-03-24T11:52:12.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:12 smithi146 ceph-mon[26954]: Upgrade: Updating mon.smithi146 2024-03-24T11:52:12.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:12 smithi146 ceph-mon[26954]: Deploying daemon mon.smithi146 on smithi146 2024-03-24T11:52:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 151465 11.17 MB/sec execute 198 sec latency 33.920 ms 2024-03-24T11:52:13.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:13 smithi032 ceph-mon[66455]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-03-24T11:52:13.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 152014 11.19 MB/sec execute 199 sec latency 61.886 ms 2024-03-24T11:52:13.481 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:13 smithi146 ceph-mon[26954]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-03-24T11:52:13.481 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:13 smithi146 systemd[1]: Stopping Ceph mon.smithi146 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:52:13.865 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146[26946]: 2024-03-24T11:52:13.480+0000 7fe924dbc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi146 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:52:13.865 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146[26946]: 2024-03-24T11:52:13.480+0000 7fe924dbc700 -1 mon.smithi146@1(peon) e2 *** Got Signal Terminated *** 2024-03-24T11:52:13.865 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:13 smithi146 podman[52845]: 2024-03-24 11:52:13.542107504 +0000 UTC m=+0.185792409 container died d29e2f462ce7638764206b1bc3cda968ee476abf2006bf796c6604332bf1c72f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream, io.openshift.expose-services=, release=754, io.buildah.version=1.19.8) 2024-03-24T11:52:14.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:13 smithi146 podman[52845]: 2024-03-24 11:52:13.981706284 +0000 UTC m=+0.625391188 container cleanup d29e2f462ce7638764206b1bc3cda968ee476abf2006bf796c6604332bf1c72f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, ceph=True, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, RELEASE=HEAD, io.buildah.version=1.19.8) 2024-03-24T11:52:14.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:13 smithi146 bash[52845]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146 2024-03-24T11:52:14.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 152586 11.26 MB/sec execute 200 sec latency 29.045 ms 2024-03-24T11:52:14.554 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:14 smithi146 podman[52860]: 2024-03-24 11:52:14.257881267 +0000 UTC m=+0.714964264 container remove d29e2f462ce7638764206b1bc3cda968ee476abf2006bf796c6604332bf1c72f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, GIT_CLEAN=True, ceph=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-24T11:52:14.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:14 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi146.service: Deactivated successfully. 2024-03-24T11:52:14.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:14 smithi146 systemd[1]: Stopped Ceph mon.smithi146 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:52:14.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:14 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi146.service: Consumed 7.036s CPU time. 2024-03-24T11:52:14.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:14 smithi146 systemd[1]: Starting Ceph mon.smithi146 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:52:15.295 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 podman[52989]: 2024-03-24 11:52:14.916738955 +0000 UTC m=+0.019283214 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:52:15.295 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 podman[52989]: 2024-03-24 11:52:15.034725036 +0000 UTC m=+0.137269277 container create 61b9704b5f2997542df34723c0a48ebe59b20dd548343dae4486db677a67bda4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T11:52:15.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 153006 11.26 MB/sec execute 201 sec latency 48.688 ms 2024-03-24T11:52:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 podman[52989]: 2024-03-24 11:52:15.295842242 +0000 UTC m=+0.398386492 container init 61b9704b5f2997542df34723c0a48ebe59b20dd548343dae4486db677a67bda4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 podman[52989]: 2024-03-24 11:52:15.299813595 +0000 UTC m=+0.402357838 container start 61b9704b5f2997542df34723c0a48ebe59b20dd548343dae4486db677a67bda4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: pidfile_write: ignore empty --pid-file 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: load: jerasure load: lrc 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Git sha 0 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: DB SUMMARY 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: DB Session ID: 1CHOWSNJYDXIU73TMW44 2024-03-24T11:52:15.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi146/store.db dir, Total Num: 1, files: 000024.sst 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi146/store.db: 000022.log size: 6929771 ; 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:52:15.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.env: 0x55983488dae0 2024-03-24T11:52:15.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:52:15.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.info_log: 0x5598370c7760 2024-03-24T11:52:15.637 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:52:15.638 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.statistics: (nil) 2024-03-24T11:52:15.638 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.use_fsync: 0 2024-03-24T11:52:15.638 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:52:15.638 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.db_log_dir: 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.wal_dir: 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:52:15.639 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.write_buffer_manager: 0x55983709b400 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:52:15.640 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.unordered_write: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.row_cache: None 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.wal_filter: None 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.wal_compression: 0 2024-03-24T11:52:15.641 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:52:15.642 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_open_files: -1 2024-03-24T11:52:15.643 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Compression algorithms supported: 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kZSTD supported: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:52:15.644 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi146/store.db/MANIFEST-000009 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.merge_operator: 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_filter: None 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5598370c7880) 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:52:15.645 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: index_type: 0 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: data_block_index_type: 0 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: index_shortening: 1 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: checksum: 4 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: no_block_cache: 0 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache: 0x5598370cb090 2024-03-24T11:52:15.646 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache_options: 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: capacity : 536870912 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: num_shard_bits : 4 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: strict_capacity_limit : 0 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache_compressed: (nil) 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: persistent_cache: (nil) 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_size: 4096 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_size_deviation: 10 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_restart_interval: 16 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: index_block_restart_interval: 1 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: metadata_block_size: 4096 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: partition_filters: 0 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: use_delta_encoding: 1 2024-03-24T11:52:15.647 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: filter_policy: bloomfilter 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: whole_key_filtering: 1 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: verify_compression: 0 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: format_version: 5 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: enable_index_compression: 1 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_align: 0 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: max_auto_readahead_size: 262144 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: prepopulate_block_cache: 0 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression: NoCompression 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:52:15.648 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.num_levels: 7 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:52:15.649 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:52:15.650 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:52:15.651 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:52:15.652 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.ttl: 2592000 2024-03-24T11:52:15.653 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:52:15.654 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi146/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12249, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d44e7ce6-3876-4581-8254-e6c8a3a699bd 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711281135344925, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711281135360297, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4213918, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12254, "largest_seqno": 13934, "table_properties": {"data_size": 4206476, "index_size": 4759, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 15371, "raw_average_key_size": 23, "raw_value_size": 4192009, "raw_average_value_size": 6303, "num_data_blocks": 216, "num_entries": 665, "num_filter_entries": 665, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711281135, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d44e7ce6-3876-4581-8254-e6c8a3a699bd", "db_session_id": "1CHOWSNJYDXIU73TMW44", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711281135360414, "job": 1, "event": "recovery_finished"} 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi146/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5598371ba000 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: DB pointer 0x5598371a6000 2024-03-24T11:52:15.655 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** DB Stats ** 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** Compaction Stats [default] ** 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: L0 1/0 4.02 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 711.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: L6 1/0 6.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-24T11:52:15.656 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Sum 2/0 10.82 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 711.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 711.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** Compaction Stats [default] ** 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 711.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:52:15.657 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative compaction: 0.00 GB write, 124.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval compaction: 0.00 GB write, 124.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Block cache BinnedLRUCache@0x5598370cb090#2 capacity: 512.00 MB usage: 56.78 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Block cache entry stats(count,size,portion): DataBlock(4,24.81 KB,0.00473261%) FilterBlock(2,10.81 KB,0.00206232%) IndexBlock(2,21.16 KB,0.00403523%) Misc(1,0.00 KB,0%) 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: starting mon.smithi146 rank 1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi146 fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???) e2 preinit fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???).mds e7 new map 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???).mds e7 print_map 2024-03-24T11:52:15.658 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: e7 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: legacy client fscid: 1 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Filesystem 'foofs' (1) 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: fs_name foofs 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: epoch 5 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: created 2024-03-24T11:46:17.024254+0000 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: modified 2024-03-24T11:46:22.041887+0000 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: tableserver 0 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: root 0 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: session_timeout 60 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: session_autoclose 300 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: max_file_size 1099511627776 2024-03-24T11:52:15.659 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: max_xattr_size 65536 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: required_client_features {} 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: last_failure 0 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: last_failure_osd_epoch 0 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: max_mds 1 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: in 0 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: up {0=14486} 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: failed 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: damaged 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: stopped 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: data_pools [3] 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: metadata_pool 2 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: inline_data disabled 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: balancer 2024-03-24T11:52:15.660 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: bal_rank_mask -1 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: standby_count_wanted 1 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: [mds.foofs.smithi032.qzdcjy{0:14486} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.32:6834/3101395579,v1:172.21.15.32:6835/3101395579] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Standby daemons: 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: [mds.foofs.smithi146.xsrsue{-1:24279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 ceph-mon[53008]: mon.smithi146@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 bash[52989]: 61b9704b5f2997542df34723c0a48ebe59b20dd548343dae4486db677a67bda4 2024-03-24T11:52:15.661 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:15 smithi146 systemd[1]: Started Ceph mon.smithi146 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:52:16.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 153304 11.23 MB/sec execute 202 sec latency 34.846 ms 2024-03-24T11:52:16.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:52:16.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: mon.smithi032 calling monitor election 2024-03-24T11:52:16.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: mon.smithi146 calling monitor election 2024-03-24T11:52:16.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: mon.smithi032 is new leader, mons smithi032,smithi146 in quorum (ranks 0,1) 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: monmap e3: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi146=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: pgmap v13: 97 pgs: 97 active+clean; 329 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 506 op/s 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: mgrmap e40: smithi032.xjdgbw(active, since 20s), standbys: smithi146.tfohfl 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: overall HEALTH_OK 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:16.790 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:16.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-03-24T11:52:16.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: mon.smithi032 calling monitor election 2024-03-24T11:52:16.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: mon.smithi146 calling monitor election 2024-03-24T11:52:16.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-03-24T11:52:16.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: mon.smithi032 is new leader, mons smithi032,smithi146 in quorum (ranks 0,1) 2024-03-24T11:52:16.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: monmap e3: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi146=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:52:16.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:52:16.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:52:16.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: pgmap v13: 97 pgs: 97 active+clean; 329 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 506 op/s 2024-03-24T11:52:16.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: mgrmap e40: smithi032.xjdgbw(active, since 20s), standbys: smithi146.tfohfl 2024-03-24T11:52:16.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: overall HEALTH_OK 2024-03-24T11:52:16.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:16.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:16.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 153868 11.24 MB/sec execute 203 sec latency 39.320 ms 2024-03-24T11:52:18.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 154483 11.24 MB/sec execute 204 sec latency 40.713 ms 2024-03-24T11:52:18.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:18 smithi032 ceph-mon[66455]: pgmap v14: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 815 op/s 2024-03-24T11:52:18.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:18 smithi146 ceph-mon[53008]: pgmap v14: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 815 op/s 2024-03-24T11:52:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 154859 11.22 MB/sec execute 205 sec latency 58.643 ms 2024-03-24T11:52:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 155272 11.24 MB/sec execute 206 sec latency 55.052 ms 2024-03-24T11:52:20.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:20 smithi032 ceph-mon[66455]: pgmap v15: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 548 op/s 2024-03-24T11:52:20.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:20 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:20.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:20 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:21.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:20 smithi146 ceph-mon[53008]: pgmap v15: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 548 op/s 2024-03-24T11:52:21.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:20 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:21.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:20 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 155585 11.23 MB/sec execute 207 sec latency 57.478 ms 2024-03-24T11:52:22.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:22 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:22.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:22 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:22.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:22 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:22.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:22 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:22.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 156023 11.28 MB/sec execute 208 sec latency 32.357 ms 2024-03-24T11:52:23.181 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:23 smithi032 ceph-mon[66455]: pgmap v16: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 548 op/s 2024-03-24T11:52:23.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:23 smithi146 ceph-mon[53008]: pgmap v16: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 548 op/s 2024-03-24T11:52:23.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 156398 11.28 MB/sec execute 209 sec latency 53.971 ms 2024-03-24T11:52:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 156810 11.25 MB/sec execute 210 sec latency 50.013 ms 2024-03-24T11:52:25.295 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:25 smithi032 ceph-mon[66455]: pgmap v17: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 8.2 MiB/s wr, 827 op/s 2024-03-24T11:52:25.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:25.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:25.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:25 smithi146 ceph-mon[53008]: pgmap v17: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 8.2 MiB/s wr, 827 op/s 2024-03-24T11:52:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:25.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 157303 11.26 MB/sec execute 211 sec latency 49.612 ms 2024-03-24T11:52:26.300 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:26 smithi032 ceph-mon[66455]: Reconfiguring mon.smithi032 (monmap changed)... 2024-03-24T11:52:26.301 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:52:26.301 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:52:26.301 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:26.301 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:26 smithi032 ceph-mon[66455]: Reconfiguring daemon mon.smithi032 on smithi032 2024-03-24T11:52:26.301 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:26.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:26 smithi146 ceph-mon[53008]: Reconfiguring mon.smithi032 (monmap changed)... 2024-03-24T11:52:26.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:52:26.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:52:26.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:26.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:26 smithi146 ceph-mon[53008]: Reconfiguring daemon mon.smithi032 on smithi032 2024-03-24T11:52:26.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:26.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 157825 11.23 MB/sec execute 212 sec latency 36.142 ms 2024-03-24T11:52:27.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:27 smithi032 ceph-mon[66455]: pgmap v18: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 589 op/s 2024-03-24T11:52:27.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:27 smithi146 ceph-mon[53008]: pgmap v18: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 589 op/s 2024-03-24T11:52:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 158288 11.24 MB/sec execute 213 sec latency 33.121 ms 2024-03-24T11:52:28.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 158834 11.27 MB/sec execute 214 sec latency 33.057 ms 2024-03-24T11:52:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 863 op/s 2024-03-24T11:52:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: Reconfiguring mgr.smithi032.xjdgbw (monmap changed)... 2024-03-24T11:52:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:52:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:52:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:29 smithi032 ceph-mon[66455]: Reconfiguring daemon mgr.smithi032.xjdgbw on smithi032 2024-03-24T11:52:29.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 159374 11.33 MB/sec execute 215 sec latency 27.836 ms 2024-03-24T11:52:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 863 op/s 2024-03-24T11:52:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: Reconfiguring mgr.smithi032.xjdgbw (monmap changed)... 2024-03-24T11:52:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.xjdgbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:52:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:52:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:29 smithi146 ceph-mon[53008]: Reconfiguring daemon mgr.smithi032.xjdgbw on smithi032 2024-03-24T11:52:29.821 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:52:30.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 159670 11.32 MB/sec execute 216 sec latency 40.192 ms 2024-03-24T11:52:30.854 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:52:30.854 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (3m) 24s ago 12m 23.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:52:30.854 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 24s ago 11m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:52:30.854 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (11m) 11s ago 11m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:52:30.854 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (2m) 24s ago 11m 53.1M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (6m) 24s ago 6m 40.5M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (6m) 11s ago 6m 16.2M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (2m) 24s ago 13m 477M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (70s) 11s ago 11m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (49s) 24s ago 13m 31.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (15s) 11s ago 11m 23.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (5m) 24s ago 5m 248M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (5m) 11s ago 5m 88.4M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (2m) 24s ago 11m 12.6M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (2m) 11s ago 11m 21.7M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 24s ago 10m 503M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 24s ago 10m 446M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 24s ago 9m 313M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (9m) 24s ago 9m 477M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (8m) 11s ago 8m 524M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:52:30.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (8m) 11s ago 8m 322M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:52:30.856 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (8m) 11s ago 8m 511M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:52:30.856 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (7m) 11s ago 7m 401M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:52:30.856 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (76s) 24s ago 10m 55.9M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:52:31.396 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:31 smithi032 ceph-mon[66455]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.1 MiB/s wr, 554 op/s 2024-03-24T11:52:31.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:31 smithi032 ceph-mon[66455]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:31.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:31 smithi032 ceph-mon[66455]: from='client.44103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:31.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 159940 11.30 MB/sec execute 217 sec latency 66.575 ms 2024-03-24T11:52:31.487 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:52:31.487 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:52:31.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:52:31.489 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:52:31.489 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-24T11:52:31.489 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:52:31.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:52:31.489 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:52:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:31 smithi146 ceph-mon[53008]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.1 MiB/s wr, 554 op/s 2024-03-24T11:52:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:31 smithi146 ceph-mon[53008]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:31 smithi146 ceph-mon[53008]: from='client.44103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:32.053 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:52:32.053 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:52:32.053 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:52:32.053 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:52:32.053 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:52:32.053 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:52:32.054 INFO:teuthology.orchestra.run.smithi032.stdout: "mon" 2024-03-24T11:52:32.054 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:52:32.054 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "4/23 daemons upgraded", 2024-03-24T11:52:32.054 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mon daemons", 2024-03-24T11:52:32.054 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:52:32.054 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:52:32.424 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:32 smithi032 ceph-mon[66455]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:32.424 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:32 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/842200272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:52:32.424 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:32 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:32.424 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:32 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:32.424 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:32 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:52:32.424 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:32 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:32.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 160174 11.27 MB/sec execute 218 sec latency 48.262 ms 2024-03-24T11:52:32.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:32 smithi146 ceph-mon[53008]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:32.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:32 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/842200272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:52:32.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:32 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:32.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:32 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:32.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:32 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:52:32.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:32 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:32.669 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:52:33.427 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:33 smithi032 ceph-mon[66455]: pgmap v21: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 582 op/s 2024-03-24T11:52:33.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:33 smithi032 ceph-mon[66455]: Reconfiguring crash.smithi032 (monmap changed)... 2024-03-24T11:52:33.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:33 smithi032 ceph-mon[66455]: Reconfiguring daemon crash.smithi032 on smithi032 2024-03-24T11:52:33.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:33 smithi032 ceph-mon[66455]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:33.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:33 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3419517666' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:52:33.449 INFO:teuthology.orchestra.run.smithi032.stdout: 5 160382 11.23 MB/sec execute 219 sec latency 46.796 ms 2024-03-24T11:52:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:33 smithi146 ceph-mon[53008]: pgmap v21: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 582 op/s 2024-03-24T11:52:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:33 smithi146 ceph-mon[53008]: Reconfiguring crash.smithi032 (monmap changed)... 2024-03-24T11:52:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:33 smithi146 ceph-mon[53008]: Reconfiguring daemon crash.smithi032 on smithi032 2024-03-24T11:52:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:33 smithi146 ceph-mon[53008]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:52:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:33 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3419517666' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:52:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 160620 11.19 MB/sec execute 220 sec latency 47.744 ms 2024-03-24T11:52:34.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:34 smithi032 ceph-mon[66455]: pgmap v22: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 801 op/s 2024-03-24T11:52:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:34 smithi032 ceph-mon[66455]: Reconfiguring osd.0 (monmap changed)... 2024-03-24T11:52:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:52:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:34 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.0 on smithi032 2024-03-24T11:52:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:34 smithi146 ceph-mon[53008]: pgmap v22: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 801 op/s 2024-03-24T11:52:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:34 smithi146 ceph-mon[53008]: Reconfiguring osd.0 (monmap changed)... 2024-03-24T11:52:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:52:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:34 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.0 on smithi032 2024-03-24T11:52:35.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 160890 11.19 MB/sec execute 221 sec latency 52.290 ms 2024-03-24T11:52:36.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 161161 11.15 MB/sec execute 222 sec latency 72.427 ms 2024-03-24T11:52:37.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:36 smithi032 ceph-mon[66455]: pgmap v23: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 520 op/s 2024-03-24T11:52:37.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:36 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:37.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:36 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:37.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:36 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:52:37.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:36 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:37.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:36 smithi146 ceph-mon[53008]: pgmap v23: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 520 op/s 2024-03-24T11:52:37.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:36 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:37.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:36 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:37.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:36 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:52:37.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:36 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:37.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 161697 11.15 MB/sec execute 223 sec latency 36.622 ms 2024-03-24T11:52:38.012 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:37 smithi032 ceph-mon[66455]: Reconfiguring osd.1 (monmap changed)... 2024-03-24T11:52:38.013 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:37 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.1 on smithi032 2024-03-24T11:52:38.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:37 smithi146 ceph-mon[53008]: Reconfiguring osd.1 (monmap changed)... 2024-03-24T11:52:38.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:37 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.1 on smithi032 2024-03-24T11:52:38.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 162249 11.17 MB/sec execute 224 sec latency 46.496 ms 2024-03-24T11:52:39.020 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:38 smithi032 ceph-mon[66455]: pgmap v24: 97 pgs: 97 active+clean; 321 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 692 op/s 2024-03-24T11:52:39.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:38 smithi146 ceph-mon[53008]: pgmap v24: 97 pgs: 97 active+clean; 321 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 692 op/s 2024-03-24T11:52:39.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 162605 11.18 MB/sec execute 225 sec latency 38.085 ms 2024-03-24T11:52:40.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 163129 11.23 MB/sec execute 226 sec latency 40.041 ms 2024-03-24T11:52:40.570 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:40.570 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:40.570 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:40 smithi032 ceph-mon[66455]: Reconfiguring osd.2 (monmap changed)... 2024-03-24T11:52:40.570 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:52:40.570 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:40.570 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:40 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.2 on smithi032 2024-03-24T11:52:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:40 smithi146 ceph-mon[53008]: Reconfiguring osd.2 (monmap changed)... 2024-03-24T11:52:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:52:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:40 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.2 on smithi032 2024-03-24T11:52:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 163545 11.23 MB/sec execute 227 sec latency 40.230 ms 2024-03-24T11:52:41.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:41 smithi032 ceph-mon[66455]: pgmap v25: 97 pgs: 97 active+clean; 321 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 418 op/s 2024-03-24T11:52:41.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:41.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:41 smithi146 ceph-mon[53008]: pgmap v25: 97 pgs: 97 active+clean; 321 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 418 op/s 2024-03-24T11:52:41.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:42.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 163944 11.21 MB/sec execute 228 sec latency 30.284 ms 2024-03-24T11:52:43.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:43 smithi032 ceph-mon[66455]: pgmap v26: 97 pgs: 97 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.3 MiB/s wr, 557 op/s 2024-03-24T11:52:43.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:43.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:43.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:43 smithi032 ceph-mon[66455]: Reconfiguring osd.3 (monmap changed)... 2024-03-24T11:52:43.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:52:43.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:43.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:43 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.3 on smithi032 2024-03-24T11:52:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:43 smithi146 ceph-mon[53008]: pgmap v26: 97 pgs: 97 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.3 MiB/s wr, 557 op/s 2024-03-24T11:52:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:43 smithi146 ceph-mon[53008]: Reconfiguring osd.3 (monmap changed)... 2024-03-24T11:52:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:52:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:43 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.3 on smithi032 2024-03-24T11:52:43.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 164421 11.21 MB/sec execute 229 sec latency 29.535 ms 2024-03-24T11:52:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 164836 11.18 MB/sec execute 230 sec latency 43.567 ms 2024-03-24T11:52:45.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:45 smithi032 ceph-mon[66455]: pgmap v27: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.7 MiB/s wr, 665 op/s 2024-03-24T11:52:45.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:45.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:45.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:52:45.103 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:45.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:45 smithi146 ceph-mon[53008]: pgmap v27: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.7 MiB/s wr, 665 op/s 2024-03-24T11:52:45.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:45.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:45.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:52:45.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 165181 11.18 MB/sec execute 231 sec latency 43.746 ms 2024-03-24T11:52:46.137 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:46 smithi032 ceph-mon[66455]: Reconfiguring mds.foofs.smithi032.qzdcjy (monmap changed)... 2024-03-24T11:52:46.138 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:46 smithi032 ceph-mon[66455]: Reconfiguring daemon mds.foofs.smithi032.qzdcjy on smithi032 2024-03-24T11:52:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:46 smithi146 ceph-mon[53008]: Reconfiguring mds.foofs.smithi032.qzdcjy (monmap changed)... 2024-03-24T11:52:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:46 smithi146 ceph-mon[53008]: Reconfiguring daemon mds.foofs.smithi032.qzdcjy on smithi032 2024-03-24T11:52:46.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 165451 11.15 MB/sec execute 232 sec latency 73.914 ms 2024-03-24T11:52:47.153 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:47 smithi032 ceph-mon[66455]: pgmap v28: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-03-24T11:52:47.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:47 smithi146 ceph-mon[53008]: pgmap v28: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-03-24T11:52:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 165847 11.17 MB/sec execute 233 sec latency 49.942 ms 2024-03-24T11:52:48.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:48 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:48.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:48 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:48.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:48 smithi032 ceph-mon[66455]: Reconfiguring crash.smithi146 (monmap changed)... 2024-03-24T11:52:48.450 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:48 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:52:48.450 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:48 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:48.450 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:48 smithi032 ceph-mon[66455]: Reconfiguring daemon crash.smithi146 on smithi146 2024-03-24T11:52:48.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 166174 11.15 MB/sec execute 234 sec latency 45.869 ms 2024-03-24T11:52:48.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:48 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:48.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:48 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:48.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:48 smithi146 ceph-mon[53008]: Reconfiguring crash.smithi146 (monmap changed)... 2024-03-24T11:52:48.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:48 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:52:48.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:48 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:48.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:48 smithi146 ceph-mon[53008]: Reconfiguring daemon crash.smithi146 on smithi146 2024-03-24T11:52:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:49 smithi146 ceph-mon[53008]: pgmap v29: 97 pgs: 97 active+clean; 322 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 696 op/s 2024-03-24T11:52:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:52:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:52:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:49.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 166578 11.19 MB/sec execute 235 sec latency 36.140 ms 2024-03-24T11:52:49.463 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:49 smithi032 ceph-mon[66455]: pgmap v29: 97 pgs: 97 active+clean; 322 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 696 op/s 2024-03-24T11:52:49.463 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:49.463 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:49.464 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.tfohfl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:52:49.464 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:52:49.464 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:50.299 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:50 smithi032 ceph-mon[66455]: Reconfiguring mgr.smithi146.tfohfl (monmap changed)... 2024-03-24T11:52:50.299 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:50 smithi032 ceph-mon[66455]: Reconfiguring daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:52:50.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 166875 11.18 MB/sec execute 236 sec latency 40.956 ms 2024-03-24T11:52:50.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:50 smithi146 ceph-mon[53008]: Reconfiguring mgr.smithi146.tfohfl (monmap changed)... 2024-03-24T11:52:50.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:50 smithi146 ceph-mon[53008]: Reconfiguring daemon mgr.smithi146.tfohfl on smithi146 2024-03-24T11:52:51.285 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:51 smithi146 ceph-mon[53008]: pgmap v30: 97 pgs: 97 active+clean; 322 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 524 op/s 2024-03-24T11:52:51.286 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:51 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:51.286 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:51 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:51.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 167169 11.16 MB/sec execute 237 sec latency 78.766 ms 2024-03-24T11:52:51.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:51 smithi032 ceph-mon[66455]: pgmap v30: 97 pgs: 97 active+clean; 322 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 524 op/s 2024-03-24T11:52:51.503 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:51 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:51.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:51 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:52.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 167363 11.13 MB/sec execute 238 sec latency 88.413 ms 2024-03-24T11:52:52.529 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:52 smithi032 ceph-mon[66455]: Reconfiguring mon.smithi146 (monmap changed)... 2024-03-24T11:52:52.530 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:52 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:52:52.530 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:52 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:52:52.530 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:52 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:52.530 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:52 smithi032 ceph-mon[66455]: Reconfiguring daemon mon.smithi146 on smithi146 2024-03-24T11:52:52.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:52 smithi146 ceph-mon[53008]: Reconfiguring mon.smithi146 (monmap changed)... 2024-03-24T11:52:52.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:52 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:52:52.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:52 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:52:52.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:52 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:52.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:52 smithi146 ceph-mon[53008]: Reconfiguring daemon mon.smithi146 on smithi146 2024-03-24T11:52:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:53 smithi146 ceph-mon[53008]: pgmap v31: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.0 MiB/s wr, 628 op/s 2024-03-24T11:52:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:53 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:53 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:53.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 167573 11.10 MB/sec execute 239 sec latency 45.247 ms 2024-03-24T11:52:53.538 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:53 smithi032 ceph-mon[66455]: pgmap v31: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.0 MiB/s wr, 628 op/s 2024-03-24T11:52:53.538 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:53 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:53.538 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:53 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:54.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:54 smithi032 ceph-mon[66455]: Reconfiguring osd.4 (monmap changed)... 2024-03-24T11:52:54.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:54 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:52:54.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:54 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:54.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:54 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.4 on smithi146 2024-03-24T11:52:54.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 167857 11.08 MB/sec execute 240 sec latency 70.847 ms 2024-03-24T11:52:54.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:54 smithi146 ceph-mon[53008]: Reconfiguring osd.4 (monmap changed)... 2024-03-24T11:52:54.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:54 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:52:54.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:54 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:54.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:54 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.4 on smithi146 2024-03-24T11:52:55.376 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:55 smithi032 ceph-mon[66455]: pgmap v32: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 574 op/s 2024-03-24T11:52:55.376 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:55.376 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:55.377 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:52:55.377 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:55 smithi146 ceph-mon[53008]: pgmap v32: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 574 op/s 2024-03-24T11:52:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:52:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:55.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 168216 11.06 MB/sec execute 241 sec latency 38.704 ms 2024-03-24T11:52:56.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:56 smithi032 ceph-mon[66455]: Reconfiguring osd.5 (monmap changed)... 2024-03-24T11:52:56.395 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:56 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.5 on smithi146 2024-03-24T11:52:56.395 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:56.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 168600 11.04 MB/sec execute 242 sec latency 49.655 ms 2024-03-24T11:52:56.621 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:56 smithi146 ceph-mon[53008]: Reconfiguring osd.5 (monmap changed)... 2024-03-24T11:52:56.622 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:56 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.5 on smithi146 2024-03-24T11:52:56.622 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:57.414 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:57 smithi032 ceph-mon[66455]: pgmap v33: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.4 MiB/s wr, 438 op/s 2024-03-24T11:52:57.415 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:57.415 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:57.415 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:52:57.415 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:57.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 168869 11.03 MB/sec execute 243 sec latency 63.757 ms 2024-03-24T11:52:57.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:57 smithi146 ceph-mon[53008]: pgmap v33: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.4 MiB/s wr, 438 op/s 2024-03-24T11:52:57.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:57.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:52:57.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:58.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 169285 11.04 MB/sec execute 244 sec latency 42.737 ms 2024-03-24T11:52:58.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:58 smithi032 ceph-mon[66455]: Reconfiguring osd.6 (monmap changed)... 2024-03-24T11:52:58.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:58 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.6 on smithi146 2024-03-24T11:52:58.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:58 smithi146 ceph-mon[53008]: Reconfiguring osd.6 (monmap changed)... 2024-03-24T11:52:58.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:58 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.6 on smithi146 2024-03-24T11:52:59.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 169593 11.02 MB/sec execute 245 sec latency 48.910 ms 2024-03-24T11:52:59.468 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:59 smithi032 ceph-mon[66455]: pgmap v34: 97 pgs: 97 active+clean; 319 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 633 op/s 2024-03-24T11:52:59.468 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:59 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:59.468 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:59 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:59.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:59 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:52:59.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:52:59 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:59.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:59 smithi146 ceph-mon[53008]: pgmap v34: 97 pgs: 97 active+clean; 319 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 633 op/s 2024-03-24T11:52:59.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:59 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:59.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:59 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:52:59.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:59 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:52:59.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:52:59 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:00.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 169870 11.03 MB/sec execute 246 sec latency 49.252 ms 2024-03-24T11:53:00.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:00 smithi032 ceph-mon[66455]: Reconfiguring osd.7 (monmap changed)... 2024-03-24T11:53:00.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:00 smithi032 ceph-mon[66455]: Reconfiguring daemon osd.7 on smithi146 2024-03-24T11:53:00.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:00 smithi146 ceph-mon[53008]: Reconfiguring osd.7 (monmap changed)... 2024-03-24T11:53:00.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:00 smithi146 ceph-mon[53008]: Reconfiguring daemon osd.7 on smithi146 2024-03-24T11:53:01.450 INFO:teuthology.orchestra.run.smithi032.stdout: 5 170129 11.04 MB/sec execute 247 sec latency 76.970 ms 2024-03-24T11:53:01.545 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:01 smithi032 ceph-mon[66455]: pgmap v35: 97 pgs: 97 active+clean; 319 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 383 op/s 2024-03-24T11:53:01.545 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:01 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:01.546 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:01 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:01.546 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:01 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:53:01.546 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:01 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:01.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:01 smithi146 ceph-mon[53008]: pgmap v35: 97 pgs: 97 active+clean; 319 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 383 op/s 2024-03-24T11:53:01.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:01 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:01.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:01 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:01.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:01 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:53:01.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:01 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:02.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:02 smithi146 ceph-mon[53008]: Reconfiguring mds.foofs.smithi146.xsrsue (monmap changed)... 2024-03-24T11:53:02.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:02 smithi146 ceph-mon[53008]: Reconfiguring daemon mds.foofs.smithi146.xsrsue on smithi146 2024-03-24T11:53:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 170345 11.01 MB/sec execute 248 sec latency 110.165 ms 2024-03-24T11:53:02.553 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:02 smithi032 ceph-mon[66455]: Reconfiguring mds.foofs.smithi146.xsrsue (monmap changed)... 2024-03-24T11:53:02.553 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:02 smithi032 ceph-mon[66455]: Reconfiguring daemon mds.foofs.smithi146.xsrsue on smithi146 2024-03-24T11:53:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:53:03.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 170593 11.00 MB/sec execute 249 sec latency 52.716 ms 2024-03-24T11:53:03.573 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: pgmap v36: 97 pgs: 97 active+clean; 318 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 393 op/s 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]: dispatch 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]': finished 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi146"}]: dispatch 2024-03-24T11:53:03.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi146"}]': finished 2024-03-24T11:53:03.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: pgmap v36: 97 pgs: 97 active+clean; 318 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 393 op/s 2024-03-24T11:53:03.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:03.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]: dispatch 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]': finished 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi146"}]: dispatch 2024-03-24T11:53:03.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi146"}]': finished 2024-03-24T11:53:04.355 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:04 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all mon 2024-03-24T11:53:04.355 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:04 smithi032 ceph-mon[66455]: from='client.34172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (3m) 58s ago 12m 23.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (12m) 58s ago 12m 6405k - 17.2.0 e1d6a67b021e 32ea4d28d353 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (11m) 44s ago 11m 6405k - 17.2.0 e1d6a67b021e 0b5a82c1c839 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 58s ago 12m 53.1M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (6m) 58s ago 6m 40.5M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (6m) 44s ago 6m 16.2M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:53:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (2m) 58s ago 13m 477M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (103s) 44s ago 11m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (82s) 58s ago 13m 31.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (49s) 44s ago 11m 23.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (6m) 58s ago 6m 248M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (6m) 44s ago 6m 88.4M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (3m) 58s ago 12m 12.6M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (2m) 44s ago 11m 21.7M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 58s ago 11m 503M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 58s ago 10m 446M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 58s ago 10m 313M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (9m) 58s ago 9m 477M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (9m) 44s ago 9m 524M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:53:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (8m) 44s ago 8m 322M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:53:04.378 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (8m) 44s ago 8m 511M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:53:04.378 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (8m) 44s ago 8m 401M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:53:04.378 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (110s) 58s ago 11m 55.9M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:53:04.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 170827 10.97 MB/sec execute 250 sec latency 50.677 ms 2024-03-24T11:53:04.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:04 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all mon 2024-03-24T11:53:04.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:04 smithi146 ceph-mon[53008]: from='client.34172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:05.039 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:53:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:53:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:53:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:53:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:53:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:53:05.041 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:53:05.368 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:05 smithi032 ceph-mon[66455]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 457 op/s 2024-03-24T11:53:05.368 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:05 smithi032 ceph-mon[66455]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:05.369 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:05 smithi032 ceph-mon[66455]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:05.369 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:05 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3055834365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 171046 10.94 MB/sec execute 251 sec latency 58.638 ms 2024-03-24T11:53:05.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:05 smithi146 ceph-mon[53008]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 457 op/s 2024-03-24T11:53:05.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:05 smithi146 ceph-mon[53008]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:05.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:05 smithi146 ceph-mon[53008]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:05.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:05 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3055834365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:05.651 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:53:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:53:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:53:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:53:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:53:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:53:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "mon" 2024-03-24T11:53:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:53:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "4/23 daemons upgraded", 2024-03-24T11:53:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading crash daemons", 2024-03-24T11:53:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:53:05.652 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:53:06.317 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:53:06.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 171278 10.91 MB/sec execute 252 sec latency 61.002 ms 2024-03-24T11:53:06.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:06 smithi032 ceph-mon[66455]: Upgrade: Updating crash.smithi032 (1/2) 2024-03-24T11:53:06.637 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:06.637 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:53:06.637 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:06.637 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:06 smithi032 ceph-mon[66455]: Deploying daemon crash.smithi032 on smithi032 2024-03-24T11:53:06.637 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:06 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/1897400350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:53:06.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:06 smithi146 ceph-mon[53008]: Upgrade: Updating crash.smithi032 (1/2) 2024-03-24T11:53:06.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:06.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:53:06.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:06.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:06 smithi146 ceph-mon[53008]: Deploying daemon crash.smithi032 on smithi032 2024-03-24T11:53:06.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:06 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/1897400350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:53:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 171595 10.91 MB/sec execute 253 sec latency 51.225 ms 2024-03-24T11:53:07.647 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:07 smithi032 ceph-mon[66455]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 372 op/s 2024-03-24T11:53:07.648 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:07 smithi032 ceph-mon[66455]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:07.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:07 smithi146 ceph-mon[53008]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 372 op/s 2024-03-24T11:53:07.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:07 smithi146 ceph-mon[53008]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:08.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 171973 10.88 MB/sec execute 254 sec latency 52.896 ms 2024-03-24T11:53:09.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 172288 10.88 MB/sec execute 255 sec latency 52.986 ms 2024-03-24T11:53:09.683 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:09 smithi032 ceph-mon[66455]: pgmap v39: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 529 op/s 2024-03-24T11:53:09.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:09 smithi146 ceph-mon[53008]: pgmap v39: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 529 op/s 2024-03-24T11:53:10.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 172521 10.85 MB/sec execute 256 sec latency 54.270 ms 2024-03-24T11:53:10.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:10 smithi032 ceph-mon[66455]: pgmap v40: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-03-24T11:53:10.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:10 smithi146 ceph-mon[53008]: pgmap v40: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-03-24T11:53:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 172948 10.87 MB/sec execute 257 sec latency 38.556 ms 2024-03-24T11:53:11.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:11.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:11.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 173288 10.87 MB/sec execute 258 sec latency 55.400 ms 2024-03-24T11:53:12.735 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:12 smithi032 ceph-mon[66455]: pgmap v41: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.2 MiB/s wr, 362 op/s 2024-03-24T11:53:12.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:12 smithi146 ceph-mon[53008]: pgmap v41: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.2 MiB/s wr, 362 op/s 2024-03-24T11:53:13.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 173668 10.89 MB/sec execute 259 sec latency 39.902 ms 2024-03-24T11:53:14.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 173973 10.88 MB/sec execute 260 sec latency 34.680 ms 2024-03-24T11:53:14.511 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:14 smithi032 ceph-mon[66455]: Upgrade: Updating crash.smithi146 (2/2) 2024-03-24T11:53:14.511 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:14 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:14.511 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:14 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:53:14.512 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:14 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:14.512 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:14 smithi032 ceph-mon[66455]: Deploying daemon crash.smithi146 on smithi146 2024-03-24T11:53:14.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:14 smithi146 ceph-mon[53008]: Upgrade: Updating crash.smithi146 (2/2) 2024-03-24T11:53:14.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:14 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:14.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:14 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:53:14.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:14 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:14.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:14 smithi146 ceph-mon[53008]: Deploying daemon crash.smithi146 on smithi146 2024-03-24T11:53:15.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 174437 10.88 MB/sec execute 261 sec latency 32.625 ms 2024-03-24T11:53:15.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:15 smithi032 ceph-mon[66455]: pgmap v42: 97 pgs: 97 active+clean; 324 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-03-24T11:53:15.822 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:15 smithi146 ceph-mon[53008]: pgmap v42: 97 pgs: 97 active+clean; 324 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-03-24T11:53:16.451 INFO:teuthology.orchestra.run.smithi032.stdout: 5 174816 10.86 MB/sec execute 262 sec latency 38.058 ms 2024-03-24T11:53:17.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 175236 10.86 MB/sec execute 263 sec latency 69.693 ms 2024-03-24T11:53:17.515 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:17 smithi146 ceph-mon[53008]: pgmap v43: 97 pgs: 97 active+clean; 324 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 358 op/s 2024-03-24T11:53:17.542 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:17 smithi032 ceph-mon[66455]: pgmap v43: 97 pgs: 97 active+clean; 324 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 358 op/s 2024-03-24T11:53:18.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 175664 10.84 MB/sec execute 264 sec latency 101.231 ms 2024-03-24T11:53:18.810 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:18 smithi032 ceph-mon[66455]: pgmap v44: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.9 MiB/s wr, 588 op/s 2024-03-24T11:53:18.810 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:18.810 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:18.810 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:19.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:18 smithi146 ceph-mon[53008]: pgmap v44: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.9 MiB/s wr, 588 op/s 2024-03-24T11:53:19.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:19.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:19.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:19.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 176071 10.84 MB/sec execute 265 sec latency 35.919 ms 2024-03-24T11:53:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 176662 10.87 MB/sec execute 266 sec latency 55.939 ms 2024-03-24T11:53:20.832 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:20 smithi032 ceph-mon[66455]: pgmap v45: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 430 op/s 2024-03-24T11:53:21.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:20 smithi146 ceph-mon[53008]: pgmap v45: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 430 op/s 2024-03-24T11:53:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 177118 10.90 MB/sec execute 267 sec latency 32.963 ms 2024-03-24T11:53:22.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 177456 10.91 MB/sec execute 268 sec latency 98.722 ms 2024-03-24T11:53:22.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:22 smithi032 ceph-mon[66455]: pgmap v46: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-03-24T11:53:22.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:22 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:22.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:22 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:23.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:22 smithi146 ceph-mon[53008]: pgmap v46: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-03-24T11:53:23.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:22 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:23.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:22 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:23.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 177866 10.91 MB/sec execute 269 sec latency 39.104 ms 2024-03-24T11:53:24.125 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:24.125 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:24.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:24.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:24.452 INFO:teuthology.orchestra.run.smithi032.stdout: 5 178139 10.88 MB/sec execute 270 sec latency 97.663 ms 2024-03-24T11:53:25.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:25 smithi032 ceph-mon[66455]: pgmap v47: 97 pgs: 97 active+clean; 330 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.5 MiB/s wr, 679 op/s 2024-03-24T11:53:25.372 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:25 smithi146 ceph-mon[53008]: pgmap v47: 97 pgs: 97 active+clean; 330 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.5 MiB/s wr, 679 op/s 2024-03-24T11:53:25.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 178529 10.87 MB/sec execute 271 sec latency 28.231 ms 2024-03-24T11:53:26.146 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:26.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:26.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 178919 10.86 MB/sec execute 272 sec latency 54.279 ms 2024-03-24T11:53:27.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:27 smithi146 ceph-mon[53008]: pgmap v48: 97 pgs: 97 active+clean; 330 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 506 op/s 2024-03-24T11:53:27.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:27 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:27.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:27 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:27.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:27 smithi032 ceph-mon[66455]: pgmap v48: 97 pgs: 97 active+clean; 330 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 506 op/s 2024-03-24T11:53:27.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:27 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:27.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:27 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 179359 10.86 MB/sec execute 273 sec latency 56.311 ms 2024-03-24T11:53:28.162 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:28 smithi032 ceph-mon[66455]: mgrmap e41: smithi032.xjdgbw(active, since 91s), standbys: smithi146.tfohfl 2024-03-24T11:53:28.162 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:28.162 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:28.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:28 smithi146 ceph-mon[53008]: mgrmap e41: smithi032.xjdgbw(active, since 91s), standbys: smithi146.tfohfl 2024-03-24T11:53:28.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:28.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:28.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 179869 10.87 MB/sec execute 274 sec latency 43.872 ms 2024-03-24T11:53:29.441 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:29 smithi032 ceph-mon[66455]: pgmap v49: 97 pgs: 97 active+clean; 324 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 743 op/s 2024-03-24T11:53:29.442 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:29.442 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:29.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 180359 10.89 MB/sec execute 275 sec latency 38.597 ms 2024-03-24T11:53:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:29 smithi146 ceph-mon[53008]: pgmap v49: 97 pgs: 97 active+clean; 324 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 743 op/s 2024-03-24T11:53:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:30.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 180780 10.92 MB/sec execute 276 sec latency 36.515 ms 2024-03-24T11:53:31.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 181274 10.93 MB/sec execute 277 sec latency 30.838 ms 2024-03-24T11:53:31.468 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:31 smithi032 ceph-mon[66455]: pgmap v50: 97 pgs: 97 active+clean; 324 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-03-24T11:53:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:31 smithi146 ceph-mon[53008]: pgmap v50: 97 pgs: 97 active+clean; 324 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-03-24T11:53:32.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 181605 10.91 MB/sec execute 278 sec latency 41.721 ms 2024-03-24T11:53:33.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 181824 10.89 MB/sec execute 279 sec latency 55.875 ms 2024-03-24T11:53:33.513 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:33 smithi032 ceph-mon[66455]: pgmap v51: 97 pgs: 97 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 622 op/s 2024-03-24T11:53:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:33 smithi146 ceph-mon[53008]: pgmap v51: 97 pgs: 97 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 622 op/s 2024-03-24T11:53:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 182092 10.87 MB/sec execute 280 sec latency 87.773 ms 2024-03-24T11:53:35.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 182379 10.84 MB/sec execute 281 sec latency 51.207 ms 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: pgmap v52: 97 pgs: 97 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 671 op/s 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.568 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]: dispatch 2024-03-24T11:53:35.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]': finished 2024-03-24T11:53:35.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi146"}]: dispatch 2024-03-24T11:53:35.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi146"}]': finished 2024-03-24T11:53:35.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:53:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: pgmap v52: 97 pgs: 97 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 671 op/s 2024-03-24T11:53:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]: dispatch 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]': finished 2024-03-24T11:53:35.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi146"}]: dispatch 2024-03-24T11:53:35.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi146"}]': finished 2024-03-24T11:53:35.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:53:36.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 182756 10.83 MB/sec execute 282 sec latency 59.957 ms 2024-03-24T11:53:36.596 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:36 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all crash 2024-03-24T11:53:36.596 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:36 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:53:36.597 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:36 smithi032 ceph-mon[66455]: Upgrade: osd.0 is safe to restart 2024-03-24T11:53:36.845 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:53:36.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:36 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all crash 2024-03-24T11:53:36.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:36 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:53:36.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:36 smithi146 ceph-mon[53008]: Upgrade: osd.0 is safe to restart 2024-03-24T11:53:37.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 183185 10.84 MB/sec execute 283 sec latency 41.778 ms 2024-03-24T11:53:37.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:37 smithi032 ceph-mon[66455]: pgmap v53: 97 pgs: 97 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-03-24T11:53:37.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:37.605 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:53:37.606 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:37 smithi146 ceph-mon[53008]: pgmap v53: 97 pgs: 97 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-03-24T11:53:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:53:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:37.967 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:53:37.967 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 10s ago 13m 23.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:53:37.967 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (27s) 10s ago 13m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:53:37.967 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (20s) 16s ago 12m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:53:37.967 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 10s ago 12m 52.1M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:53:37.967 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (7m) 10s ago 7m 40.9M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:53:37.967 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (7m) 16s ago 7m 16.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (3m) 10s ago 14m 496M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (2m) 16s ago 12m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (116s) 10s ago 14m 43.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (83s) 16s ago 12m 33.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (6m) 10s ago 6m 257M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (6m) 16s ago 6m 88.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (3m) 10s ago 12m 16.3M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (3m) 16s ago 12m 20.1M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 10s ago 11m 359M 1937M 17.2.0 e1d6a67b021e c72206819bc4 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 10s ago 11m 543M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 10s ago 10m 397M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (10m) 10s ago 10m 540M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:53:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (9m) 16s ago 9m 566M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:53:37.969 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (9m) 16s ago 9m 397M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:53:37.969 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (9m) 16s ago 9m 575M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:53:37.969 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (8m) 16s ago 8m 498M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:53:37.969 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (2m) 10s ago 12m 66.4M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:53:38.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 183774 10.86 MB/sec execute 284 sec latency 40.187 ms 2024-03-24T11:53:38.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:38 smithi032 ceph-mon[66455]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:38.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:38 smithi032 ceph-mon[66455]: Upgrade: Updating osd.0 2024-03-24T11:53:38.616 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:38 smithi032 ceph-mon[66455]: Deploying daemon osd.0 on smithi032 2024-03-24T11:53:38.617 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:38 smithi032 ceph-mon[66455]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:53:38.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:53:38.618 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:53:38.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:38 smithi146 ceph-mon[53008]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:38.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:38 smithi146 ceph-mon[53008]: Upgrade: Updating osd.0 2024-03-24T11:53:38.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:38 smithi146 ceph-mon[53008]: Deploying daemon osd.0 on smithi032 2024-03-24T11:53:38.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:38 smithi146 ceph-mon[53008]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:39.127 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:53:39.127 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:53:39.127 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:53:39.127 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "6/23 daemons upgraded", 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:53:39.128 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:53:39.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 184125 10.88 MB/sec execute 285 sec latency 39.095 ms 2024-03-24T11:53:39.635 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:39 smithi032 ceph-mon[66455]: pgmap v54: 97 pgs: 97 active+clean; 315 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 710 op/s 2024-03-24T11:53:39.635 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:39 smithi032 ceph-mon[66455]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:39.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:39 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/1940971944' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:39.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:39 smithi032 ceph-mon[66455]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:39.796 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:53:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:39 smithi146 ceph-mon[53008]: pgmap v54: 97 pgs: 97 active+clean; 315 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 710 op/s 2024-03-24T11:53:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:39 smithi146 ceph-mon[53008]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:39.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:39 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/1940971944' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:53:39.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:39 smithi146 ceph-mon[53008]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:53:40.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 184390 10.88 MB/sec execute 286 sec latency 82.171 ms 2024-03-24T11:53:40.662 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:40 smithi032 ceph-mon[66455]: pgmap v55: 97 pgs: 97 active+clean; 315 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 473 op/s 2024-03-24T11:53:40.662 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:40 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/606589871' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:53:40.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:40 smithi146 ceph-mon[53008]: pgmap v55: 97 pgs: 97 active+clean; 315 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 473 op/s 2024-03-24T11:53:40.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:40 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/606589871' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:53:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 184700 10.86 MB/sec execute 287 sec latency 63.649 ms 2024-03-24T11:53:41.686 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:41.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:42.199 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:42 smithi032 systemd[1]: Stopping Ceph osd.0 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:53:42.452 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:42 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[35181]: 2024-03-24T11:53:42.413+0000 7fcdebf2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:53:42.452 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:42 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[35181]: 2024-03-24T11:53:42.413+0000 7fcdebf2f700 -1 osd.0 72 *** Got signal Terminated *** 2024-03-24T11:53:42.453 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:42 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[35181]: 2024-03-24T11:53:42.413+0000 7fcdebf2f700 -1 osd.0 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:53:42.453 INFO:teuthology.orchestra.run.smithi032.stdout: 5 185042 10.85 MB/sec execute 288 sec latency 53.252 ms 2024-03-24T11:53:42.713 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:42 smithi032 ceph-mon[66455]: pgmap v56: 97 pgs: 97 active+clean; 322 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.4 MiB/s wr, 604 op/s 2024-03-24T11:53:42.713 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:42 smithi032 ceph-mon[66455]: osd.0 marked itself down and dead 2024-03-24T11:53:42.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:42 smithi146 ceph-mon[53008]: pgmap v56: 97 pgs: 97 active+clean; 322 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.4 MiB/s wr, 604 op/s 2024-03-24T11:53:42.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:42 smithi146 ceph-mon[53008]: osd.0 marked itself down and dead 2024-03-24T11:53:43.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 185207 10.82 MB/sec execute 289 sec latency 851.932 ms 2024-03-24T11:53:43.601 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:43 smithi032 podman[72109]: 2024-03-24 11:53:43.229652524 +0000 UTC m=+0.947916349 container died c72206819bc4a8fbdfa4e39e25b515301c9800f7d4b9d11fea47fb9bd1611946 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, distribution-scope=public, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, release=754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, ceph=True, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-03-24T11:53:43.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:43 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:53:43.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:43 smithi032 ceph-mon[66455]: failed to encode map e73 with expected crc 2024-03-24T11:53:43.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:43 smithi032 ceph-mon[66455]: osdmap e73: 8 total, 7 up, 8 in 2024-03-24T11:53:43.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:43 smithi032 ceph-mon[66455]: failed to encode map e73 with expected crc 2024-03-24T11:53:43.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:43 smithi032 ceph-mon[66455]: failed to encode map e73 with expected crc 2024-03-24T11:53:43.855 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:43 smithi032 podman[72109]: 2024-03-24 11:53:43.761583505 +0000 UTC m=+1.479847329 container cleanup c72206819bc4a8fbdfa4e39e25b515301c9800f7d4b9d11fea47fb9bd1611946 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, ceph=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.component=centos-stream-container, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, version=8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64) 2024-03-24T11:53:43.855 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:43 smithi032 bash[72109]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:53:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:43 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:53:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:43 smithi146 ceph-mon[53008]: failed to encode map e73 with expected crc 2024-03-24T11:53:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:43 smithi146 ceph-mon[53008]: osdmap e73: 8 total, 7 up, 8 in 2024-03-24T11:53:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:43 smithi146 ceph-mon[53008]: failed to encode map e73 with expected crc 2024-03-24T11:53:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:43 smithi146 ceph-mon[53008]: failed to encode map e73 with expected crc 2024-03-24T11:53:44.105 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:44 smithi032 podman[72124]: 2024-03-24 11:53:44.093343986 +0000 UTC m=+0.860215587 container remove c72206819bc4a8fbdfa4e39e25b515301c9800f7d4b9d11fea47fb9bd1611946 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, architecture=x86_64, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, name=centos-stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-type=git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-24T11:53:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 185474 10.81 MB/sec execute 290 sec latency 1039.215 ms 2024-03-24T11:53:44.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: osdmap e74: 8 total, 7 up, 8 in 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:44 smithi032 ceph-mon[66455]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 629 op/s 2024-03-24T11:53:44.862 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:44 smithi032 podman[72208]: 2024-03-24 11:53:44.589562266 +0000 UTC m=+0.024038864 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:53:44.863 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:44 smithi032 podman[72208]: 2024-03-24 11:53:44.753741045 +0000 UTC m=+0.188217638 container create 70b6308785a24adc042d40f09f8d9facb653e255f495a669e2982c101c7ba0c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-24T11:53:44.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: osdmap e74: 8 total, 7 up, 8 in 2024-03-24T11:53:44.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: failed to encode map e74 with expected crc 2024-03-24T11:53:44.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:44 smithi146 ceph-mon[53008]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 629 op/s 2024-03-24T11:53:45.126 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:45 smithi032 podman[72208]: 2024-03-24 11:53:45.095298969 +0000 UTC m=+0.529775574 container init 70b6308785a24adc042d40f09f8d9facb653e255f495a669e2982c101c7ba0c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T11:53:45.127 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:45 smithi032 podman[72208]: 2024-03-24 11:53:45.1073271 +0000 UTC m=+0.541803692 container start 70b6308785a24adc042d40f09f8d9facb653e255f495a669e2982c101c7ba0c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-24T11:53:45.380 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:45 smithi032 podman[72208]: 2024-03-24 11:53:45.245880699 +0000 UTC m=+0.680357300 container attach 70b6308785a24adc042d40f09f8d9facb653e255f495a669e2982c101c7ba0c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T11:53:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 185888 10.79 MB/sec execute 291 sec latency 55.704 ms 2024-03-24T11:53:45.634 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:45 smithi032 podman[72208]: 2024-03-24 11:53:45.569338895 +0000 UTC m=+1.003815505 container died 70b6308785a24adc042d40f09f8d9facb653e255f495a669e2982c101c7ba0c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True) 2024-03-24T11:53:46.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 186365 10.79 MB/sec execute 292 sec latency 61.378 ms 2024-03-24T11:53:46.644 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:46 smithi032 podman[72233]: 2024-03-24 11:53:46.522194472 +0000 UTC m=+0.945150003 container remove 70b6308785a24adc042d40f09f8d9facb653e255f495a669e2982c101c7ba0c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T11:53:46.644 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:46 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T11:53:46.644 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:46 smithi032 systemd[1]: Stopped Ceph osd.0 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:53:46.644 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:46 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.0.service: Consumed 3min 14.486s CPU time. 2024-03-24T11:53:46.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:46 smithi032 ceph-mon[66455]: pgmap v60: 97 pgs: 13 stale+active+clean, 84 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.0 MiB/s wr, 329 op/s 2024-03-24T11:53:46.895 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:46 smithi032 systemd[1]: Starting Ceph osd.0 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:53:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:46 smithi146 ceph-mon[53008]: pgmap v60: 97 pgs: 13 stale+active+clean, 84 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.0 MiB/s wr, 329 op/s 2024-03-24T11:53:47.400 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:47 smithi032 podman[72329]: 2024-03-24 11:53:47.063120403 +0000 UTC m=+0.027037143 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:53:47.400 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:47 smithi032 podman[72329]: 2024-03-24 11:53:47.199113361 +0000 UTC m=+0.163030090 container create 63b791966d7fda901129248a266e509f093a1e709ac12f292c91ba7f72f4bcf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:53:47.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 186948 10.81 MB/sec execute 293 sec latency 40.253 ms 2024-03-24T11:53:47.655 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:47 smithi032 podman[72329]: 2024-03-24 11:53:47.613729733 +0000 UTC m=+0.577646462 container init 63b791966d7fda901129248a266e509f093a1e709ac12f292c91ba7f72f4bcf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:53:47.655 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:47 smithi032 podman[72329]: 2024-03-24 11:53:47.623653739 +0000 UTC m=+0.587570463 container start 63b791966d7fda901129248a266e509f093a1e709ac12f292c91ba7f72f4bcf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-24T11:53:47.906 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:47 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 205/1566 objects degraded (13.091%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:53:47.906 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:47 smithi032 podman[72329]: 2024-03-24 11:53:47.84575369 +0000 UTC m=+0.809670435 container attach 63b791966d7fda901129248a266e509f093a1e709ac12f292c91ba7f72f4bcf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-24T11:53:47.973 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:47 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 205/1566 objects degraded (13.091%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate[72343]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate[72343]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 bash[72329]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 bash[72329]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 bash[72329]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 bash[72329]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 bash[72329]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 bash[72329]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 bash[72329]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T11:53:48.413 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate[72343]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:53:48.414 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate[72343]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:53:48.414 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate[72343]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T11:53:48.414 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate[72343]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:53:48.414 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate[72343]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T11:53:48.414 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:48 smithi032 podman[72329]: 2024-03-24 11:53:48.266094805 +0000 UTC m=+1.230011528 container died 63b791966d7fda901129248a266e509f093a1e709ac12f292c91ba7f72f4bcf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T11:53:48.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 187418 10.83 MB/sec execute 294 sec latency 33.788 ms 2024-03-24T11:53:48.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:48 smithi032 ceph-mon[66455]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 323 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.6 MiB/s wr, 665 op/s; 205/1566 objects degraded (13.091%) 2024-03-24T11:53:49.062 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:48 smithi146 ceph-mon[53008]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 323 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.6 MiB/s wr, 665 op/s; 205/1566 objects degraded (13.091%) 2024-03-24T11:53:49.424 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:49 smithi032 podman[72475]: 2024-03-24 11:53:49.220475872 +0000 UTC m=+0.945001294 container remove 63b791966d7fda901129248a266e509f093a1e709ac12f292c91ba7f72f4bcf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T11:53:49.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 187978 10.87 MB/sec execute 295 sec latency 25.981 ms 2024-03-24T11:53:49.676 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:49 smithi032 podman[72531]: 2024-03-24 11:53:49.394581934 +0000 UTC m=+0.025051228 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:53:49.676 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:49 smithi032 podman[72531]: 2024-03-24 11:53:49.567353648 +0000 UTC m=+0.197822928 container create 7ec0013a993cc22681fee213e0e548c539e0b29bf85297130ff21e08d95db5f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:53:50.021 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:49 smithi032 podman[72531]: 2024-03-24 11:53:49.942619041 +0000 UTC m=+0.573088343 container init 7ec0013a993cc22681fee213e0e548c539e0b29bf85297130ff21e08d95db5f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T11:53:50.021 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:49 smithi032 podman[72531]: 2024-03-24 11:53:49.9475995 +0000 UTC m=+0.578068781 container start 7ec0013a993cc22681fee213e0e548c539e0b29bf85297130ff21e08d95db5f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T11:53:50.021 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:49 smithi032 ceph-osd[72581]: -- 172.21.15.32:0/3815808226 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561a63a171e0 con 0x561a639e9000 2024-03-24T11:53:50.184 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:50 smithi032 bash[72531]: 7ec0013a993cc22681fee213e0e548c539e0b29bf85297130ff21e08d95db5f2 2024-03-24T11:53:50.184 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:50 smithi032 systemd[1]: Started Ceph osd.0 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:53:50.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 188407 10.88 MB/sec execute 296 sec latency 45.502 ms 2024-03-24T11:53:50.700 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:50 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[72577]: 2024-03-24T11:53:50.532+0000 7fe2ac5567c0 -1 Falling back to public interface 2024-03-24T11:53:51.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:51 smithi032 ceph-mon[66455]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 323 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.0 MiB/s wr, 468 op/s; 205/1566 objects degraded (13.091%) 2024-03-24T11:53:51.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:51 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:51.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:51 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:51.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:51 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:51.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 188559 10.85 MB/sec execute 297 sec latency 70.354 ms 2024-03-24T11:53:51.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:51 smithi146 ceph-mon[53008]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 323 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.0 MiB/s wr, 468 op/s; 205/1566 objects degraded (13.091%) 2024-03-24T11:53:51.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:51 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:51.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:51 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:51.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:51 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 188771 10.82 MB/sec execute 298 sec latency 62.861 ms 2024-03-24T11:53:53.264 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:53 smithi032 ceph-mon[66455]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.9 MiB/s wr, 490 op/s; 219/1638 objects degraded (13.370%) 2024-03-24T11:53:53.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 188981 10.80 MB/sec execute 299 sec latency 53.903 ms 2024-03-24T11:53:53.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:53 smithi146 ceph-mon[53008]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.9 MiB/s wr, 490 op/s; 219/1638 objects degraded (13.370%) 2024-03-24T11:53:54.297 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:54 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 230/1707 objects degraded (13.474%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:53:54.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 189247 10.78 MB/sec execute 300 sec latency 62.485 ms 2024-03-24T11:53:54.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:54 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 230/1707 objects degraded (13.474%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:53:55.305 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:55 smithi032 ceph-mon[66455]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.7 MiB/s wr, 571 op/s; 230/1707 objects degraded (13.474%) 2024-03-24T11:53:55.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 189590 10.76 MB/sec execute 301 sec latency 54.774 ms 2024-03-24T11:53:55.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:55 smithi146 ceph-mon[53008]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.7 MiB/s wr, 571 op/s; 230/1707 objects degraded (13.474%) 2024-03-24T11:53:56.322 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:56 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[72577]: 2024-03-24T11:53:56.277+0000 7fe2ac5567c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-24T11:53:56.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 189872 10.75 MB/sec execute 302 sec latency 62.671 ms 2024-03-24T11:53:56.828 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:56 smithi032 ceph-mon[66455]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 477 op/s; 230/1707 objects degraded (13.474%) 2024-03-24T11:53:56.828 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:56.829 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:56 smithi146 ceph-mon[53008]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 477 op/s; 230/1707 objects degraded (13.474%) 2024-03-24T11:53:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:53:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:57.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 190154 10.73 MB/sec execute 303 sec latency 47.503 ms 2024-03-24T11:53:57.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:57 smithi032 ceph-mon[66455]: from='osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:53:57.840 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:57 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[72577]: 2024-03-24T11:53:57.680+0000 7fe2ac5567c0 -1 osd.0 72 log_to_monitors true 2024-03-24T11:53:57.840 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 11:53:57 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[72577]: 2024-03-24T11:53:57.757+0000 7fe29d9ae700 -1 osd.0 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:53:58.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:57 smithi146 ceph-mon[53008]: from='osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:53:58.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 190502 10.74 MB/sec execute 304 sec latency 63.923 ms 2024-03-24T11:53:58.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:58 smithi032 ceph-mon[66455]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 324 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.1 MiB/s wr, 653 op/s; 198/1518 objects degraded (13.043%) 2024-03-24T11:53:58.918 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:58 smithi032 ceph-mon[66455]: from='osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:53:58.919 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:58 smithi032 ceph-mon[66455]: failed to encode map e75 with expected crc 2024-03-24T11:53:58.919 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:58 smithi032 ceph-mon[66455]: osdmap e75: 8 total, 7 up, 8 in 2024-03-24T11:53:58.919 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:58 smithi032 ceph-mon[66455]: from='osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:53:58.919 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:58 smithi032 ceph-mon[66455]: failed to encode map e75 with expected crc 2024-03-24T11:53:59.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:58 smithi146 ceph-mon[53008]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 324 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.1 MiB/s wr, 653 op/s; 198/1518 objects degraded (13.043%) 2024-03-24T11:53:59.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:58 smithi146 ceph-mon[53008]: from='osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:53:59.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:58 smithi146 ceph-mon[53008]: failed to encode map e75 with expected crc 2024-03-24T11:53:59.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:58 smithi146 ceph-mon[53008]: osdmap e75: 8 total, 7 up, 8 in 2024-03-24T11:53:59.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:58 smithi146 ceph-mon[53008]: from='osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:53:59.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:58 smithi146 ceph-mon[53008]: failed to encode map e75 with expected crc 2024-03-24T11:53:59.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 190560 10.71 MB/sec execute 305 sec latency 742.104 ms 2024-03-24T11:54:00.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:54:00.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: failed to encode map e76 with expected crc 2024-03-24T11:54:00.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936] boot 2024-03-24T11:54:00.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:54:00.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:54:00.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: failed to encode map e76 with expected crc 2024-03-24T11:54:00.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:00.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:53:59 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:00.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:54:00.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: failed to encode map e76 with expected crc 2024-03-24T11:54:00.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: osd.0 [v2:172.21.15.32:6802/3762285936,v1:172.21.15.32:6803/3762285936] boot 2024-03-24T11:54:00.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:54:00.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:54:00.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: failed to encode map e76 with expected crc 2024-03-24T11:54:00.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:00.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:53:59 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 190560 10.67 MB/sec execute 306 sec latency 1742.148 ms 2024-03-24T11:54:01.029 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 324 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.6 MiB/s wr, 427 op/s; 198/1518 objects degraded (13.043%) 2024-03-24T11:54:01.029 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:01.030 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:01.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 324 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.6 MiB/s wr, 427 op/s; 198/1518 objects degraded (13.043%) 2024-03-24T11:54:01.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:54:01.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: failed to encode map e77 with expected crc 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:01.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:01.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 190715 10.65 MB/sec execute 307 sec latency 2343.876 ms 2024-03-24T11:54:01.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:01 smithi032 ceph-mon[66455]: failed to encode map e78 with expected crc 2024-03-24T11:54:01.781 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:01 smithi032 ceph-mon[66455]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:54:01.781 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:01 smithi032 ceph-mon[66455]: failed to encode map e78 with expected crc 2024-03-24T11:54:01.781 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:01 smithi032 ceph-mon[66455]: failed to encode map e78 with expected crc 2024-03-24T11:54:01.781 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:01 smithi032 ceph-mon[66455]: failed to encode map e78 with expected crc 2024-03-24T11:54:01.781 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:01 smithi032 ceph-mon[66455]: failed to encode map e78 with expected crc 2024-03-24T11:54:02.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:01 smithi146 ceph-mon[53008]: failed to encode map e78 with expected crc 2024-03-24T11:54:02.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:01 smithi146 ceph-mon[53008]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:54:02.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:01 smithi146 ceph-mon[53008]: failed to encode map e78 with expected crc 2024-03-24T11:54:02.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:01 smithi146 ceph-mon[53008]: failed to encode map e78 with expected crc 2024-03-24T11:54:02.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:01 smithi146 ceph-mon[53008]: failed to encode map e78 with expected crc 2024-03-24T11:54:02.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:01 smithi146 ceph-mon[53008]: failed to encode map e78 with expected crc 2024-03-24T11:54:02.454 INFO:teuthology.orchestra.run.smithi032.stdout: 5 191228 10.68 MB/sec execute 308 sec latency 32.275 ms 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: pgmap v72: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 5 active+undersized, 72 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s wr, 88 op/s; 202/1515 objects degraded (13.333%); 22 B/s, 7 objects/s recovering 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 202/1515 objects degraded (13.333%), 20 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: failed to encode map e79 with expected crc 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: failed to encode map e79 with expected crc 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: failed to encode map e79 with expected crc 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: failed to encode map e80 with expected crc 2024-03-24T11:54:02.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:02 smithi032 ceph-mon[66455]: failed to encode map e80 with expected crc 2024-03-24T11:54:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: pgmap v72: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 5 active+undersized, 72 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s wr, 88 op/s; 202/1515 objects degraded (13.333%); 22 B/s, 7 objects/s recovering 2024-03-24T11:54:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 202/1515 objects degraded (13.333%), 20 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: failed to encode map e79 with expected crc 2024-03-24T11:54:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:54:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: failed to encode map e79 with expected crc 2024-03-24T11:54:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: failed to encode map e79 with expected crc 2024-03-24T11:54:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:54:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: failed to encode map e80 with expected crc 2024-03-24T11:54:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:02 smithi146 ceph-mon[53008]: failed to encode map e80 with expected crc 2024-03-24T11:54:03.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 191598 10.69 MB/sec execute 309 sec latency 830.370 ms 2024-03-24T11:54:03.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e80 with expected crc 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e80 with expected crc 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e80 with expected crc 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e80 with expected crc 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:03.795 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:03 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e80 with expected crc 2024-03-24T11:54:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e80 with expected crc 2024-03-24T11:54:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e80 with expected crc 2024-03-24T11:54:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e80 with expected crc 2024-03-24T11:54:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:54:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:03 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 191885 10.68 MB/sec execute 310 sec latency 875.287 ms 2024-03-24T11:54:04.808 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.808 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.808 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e81 with expected crc 2024-03-24T11:54:04.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: pgmap v76: 97 pgs: 1 peering, 88 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 327 op/s; 116/1530 objects degraded (7.582%); 2.1 MiB/s, 30 objects/s recovering 2024-03-24T11:54:04.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:54:04.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:04 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:05.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e81 with expected crc 2024-03-24T11:54:05.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: pgmap v76: 97 pgs: 1 peering, 88 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 327 op/s; 116/1530 objects degraded (7.582%); 2.1 MiB/s, 30 objects/s recovering 2024-03-24T11:54:05.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:54:05.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:04 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 192132 10.67 MB/sec execute 311 sec latency 831.575 ms 2024-03-24T11:54:05.830 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:05 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:05 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:05 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:05 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:05 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:05 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:05.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:05 smithi032 ceph-mon[66455]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:05 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:05 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:05 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:05 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:05 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:05 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:05 smithi146 ceph-mon[53008]: failed to encode map e82 with expected crc 2024-03-24T11:54:06.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 192427 10.66 MB/sec execute 312 sec latency 34.988 ms 2024-03-24T11:54:06.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: pgmap v78: 97 pgs: 1 peering, 88 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.8 MiB/s wr, 230 op/s; 116/1530 objects degraded (7.582%); 2.0 MiB/s, 22 objects/s recovering 2024-03-24T11:54:06.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:06.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:06 smithi032 ceph-mon[66455]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:54:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: pgmap v78: 97 pgs: 1 peering, 88 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.8 MiB/s wr, 230 op/s; 116/1530 objects degraded (7.582%); 2.0 MiB/s, 22 objects/s recovering 2024-03-24T11:54:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:07.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:06 smithi146 ceph-mon[53008]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:54:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 192878 10.64 MB/sec execute 313 sec latency 58.953 ms 2024-03-24T11:54:07.876 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:07 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 116/1530 objects degraded (7.582%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:08.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:07 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 116/1530 objects degraded (7.582%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:08.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 193254 10.64 MB/sec execute 314 sec latency 38.407 ms 2024-03-24T11:54:08.909 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:08 smithi032 ceph-mon[66455]: pgmap v79: 97 pgs: 1 peering, 94 active+clean, 2 active+recovery_wait+undersized+degraded+remapped; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.9 MiB/s wr, 386 op/s; 47/1608 objects degraded (2.923%); 5 B/s, 21 objects/s recovering 2024-03-24T11:54:09.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:08 smithi146 ceph-mon[53008]: pgmap v79: 97 pgs: 1 peering, 94 active+clean, 2 active+recovery_wait+undersized+degraded+remapped; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.9 MiB/s wr, 386 op/s; 47/1608 objects degraded (2.923%); 5 B/s, 21 objects/s recovering 2024-03-24T11:54:09.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 193723 10.65 MB/sec execute 315 sec latency 37.557 ms 2024-03-24T11:54:09.927 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:09 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47/1608 objects degraded (2.923%), 2 pgs degraded) 2024-03-24T11:54:09.927 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:09 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:54:10.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:09 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47/1608 objects degraded (2.923%), 2 pgs degraded) 2024-03-24T11:54:10.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:09 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:54:10.368 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:54:10.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 194238 10.65 MB/sec execute 316 sec latency 59.989 ms 2024-03-24T11:54:10.936 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:10 smithi032 ceph-mon[66455]: pgmap v80: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.0 MiB/s wr, 346 op/s; 4 B/s, 27 objects/s recovering 2024-03-24T11:54:10.936 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:10 smithi032 ceph-mon[66455]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:10.936 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:10.936 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:10 smithi146 ceph-mon[53008]: pgmap v80: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.0 MiB/s wr, 346 op/s; 4 B/s, 27 objects/s recovering 2024-03-24T11:54:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:10 smithi146 ceph-mon[53008]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:11.359 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:54:11.359 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 12s ago 13m 22.5M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:54:11.359 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (60s) 12s ago 13m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:54:11.359 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (54s) 49s ago 13m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:54:11.359 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (4m) 12s ago 13m 54.5M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:54:11.359 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (7m) 12s ago 7m 40.5M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (7m) 49s ago 7m 16.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (3m) 12s ago 14m 502M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (2m) 49s ago 13m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (2m) 12s ago 15m 45.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (116s) 49s ago 12m 33.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (7m) 12s ago 7m 253M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (7m) 49s ago 7m 88.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (4m) 12s ago 13m 16.2M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (3m) 49s ago 12m 20.1M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (21s) 12s ago 12m 15.1M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 12s ago 11m 575M 1937M 17.2.0 e1d6a67b021e adc99f43ac48 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 12s ago 11m 447M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (10m) 12s ago 10m 585M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (10m) 49s ago 10m 566M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:54:11.361 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (10m) 49s ago 10m 397M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:54:11.361 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (9m) 49s ago 9m 575M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:54:11.361 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (9m) 49s ago 9m 498M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:54:11.361 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (2m) 12s ago 12m 66.6M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:54:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 194606 10.66 MB/sec execute 317 sec latency 36.159 ms 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:54:11.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:54:11.934 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:54:11.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:11 smithi032 ceph-mon[66455]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:11.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:11 smithi032 ceph-mon[66455]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:12.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:11 smithi146 ceph-mon[53008]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:12.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:11 smithi146 ceph-mon[53008]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:12.416 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "7/23 daemons upgraded", 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:54:12.417 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:54:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 195007 10.67 MB/sec execute 318 sec latency 36.167 ms 2024-03-24T11:54:12.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:12 smithi032 ceph-mon[66455]: pgmap v81: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.3 MiB/s wr, 530 op/s; 0 B/s, 17 objects/s recovering 2024-03-24T11:54:12.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:12 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3896825600' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:12.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:12 smithi032 ceph-mon[66455]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:13.091 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:54:13.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:12 smithi146 ceph-mon[53008]: pgmap v81: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.3 MiB/s wr, 530 op/s; 0 B/s, 17 objects/s recovering 2024-03-24T11:54:13.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:12 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3896825600' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:13.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:12 smithi146 ceph-mon[53008]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:13.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 195471 10.68 MB/sec execute 319 sec latency 35.680 ms 2024-03-24T11:54:14.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:13 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/710159167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:54:14.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:13 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/710159167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:54:14.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 195819 10.67 MB/sec execute 320 sec latency 36.834 ms 2024-03-24T11:54:15.026 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:14 smithi032 ceph-mon[66455]: pgmap v82: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.8 MiB/s wr, 566 op/s; 0 B/s, 14 objects/s recovering 2024-03-24T11:54:15.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:14 smithi146 ceph-mon[53008]: pgmap v82: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.8 MiB/s wr, 566 op/s; 0 B/s, 14 objects/s recovering 2024-03-24T11:54:15.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 196216 10.66 MB/sec execute 321 sec latency 53.779 ms 2024-03-24T11:54:16.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 196578 10.65 MB/sec execute 322 sec latency 45.744 ms 2024-03-24T11:54:16.808 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:16 smithi032 ceph-mon[66455]: pgmap v83: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.2 MiB/s wr, 507 op/s; 0 B/s, 12 objects/s recovering 2024-03-24T11:54:16.808 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:17.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:16 smithi146 ceph-mon[53008]: pgmap v83: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.2 MiB/s wr, 507 op/s; 0 B/s, 12 objects/s recovering 2024-03-24T11:54:17.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:17.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 196801 10.63 MB/sec execute 323 sec latency 43.578 ms 2024-03-24T11:54:18.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 197262 10.66 MB/sec execute 324 sec latency 51.452 ms 2024-03-24T11:54:18.873 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:18 smithi032 ceph-mon[66455]: pgmap v84: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.6 MiB/s wr, 617 op/s; 0 B/s, 11 objects/s recovering 2024-03-24T11:54:19.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:18 smithi146 ceph-mon[53008]: pgmap v84: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.6 MiB/s wr, 617 op/s; 0 B/s, 11 objects/s recovering 2024-03-24T11:54:19.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 197767 10.64 MB/sec execute 325 sec latency 43.947 ms 2024-03-24T11:54:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 198077 10.65 MB/sec execute 326 sec latency 40.993 ms 2024-03-24T11:54:20.906 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:20 smithi032 ceph-mon[66455]: pgmap v85: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.4 MiB/s wr, 525 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:54:21.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:20 smithi146 ceph-mon[53008]: pgmap v85: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.4 MiB/s wr, 525 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:54:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 198388 10.66 MB/sec execute 327 sec latency 45.603 ms 2024-03-24T11:54:21.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:21 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:21.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:21 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:21.923 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:21 smithi032 ceph-mon[66455]: Upgrade: osd.1 is safe to restart 2024-03-24T11:54:22.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:21 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:22.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:21 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:54:22.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:21 smithi146 ceph-mon[53008]: Upgrade: osd.1 is safe to restart 2024-03-24T11:54:22.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 198717 10.65 MB/sec execute 328 sec latency 82.331 ms 2024-03-24T11:54:22.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:22 smithi032 ceph-mon[66455]: pgmap v86: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.4 MiB/s wr, 697 op/s 2024-03-24T11:54:23.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:22 smithi146 ceph-mon[53008]: pgmap v86: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.4 MiB/s wr, 697 op/s 2024-03-24T11:54:23.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 198987 10.64 MB/sec execute 329 sec latency 60.119 ms 2024-03-24T11:54:24.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 199202 10.63 MB/sec execute 330 sec latency 45.548 ms 2024-03-24T11:54:24.778 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:24 smithi032 ceph-mon[66455]: pgmap v87: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 573 op/s 2024-03-24T11:54:24.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:24 smithi032 ceph-mon[66455]: Upgrade: Updating osd.1 2024-03-24T11:54:24.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:24.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:54:24.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:24.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:24 smithi032 ceph-mon[66455]: Deploying daemon osd.1 on smithi032 2024-03-24T11:54:25.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:24 smithi146 ceph-mon[53008]: pgmap v87: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 573 op/s 2024-03-24T11:54:25.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:24 smithi146 ceph-mon[53008]: Upgrade: Updating osd.1 2024-03-24T11:54:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:54:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:25.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:24 smithi146 ceph-mon[53008]: Deploying daemon osd.1 on smithi032 2024-03-24T11:54:25.456 INFO:teuthology.orchestra.run.smithi032.stdout: 5 199566 10.63 MB/sec execute 331 sec latency 56.077 ms 2024-03-24T11:54:26.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 200242 10.62 MB/sec execute 332 sec latency 18.887 ms 2024-03-24T11:54:26.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:26 smithi032 ceph-mon[66455]: pgmap v88: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 462 op/s 2024-03-24T11:54:26.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:27.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:26 smithi146 ceph-mon[53008]: pgmap v88: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 462 op/s 2024-03-24T11:54:27.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 200894 10.66 MB/sec execute 333 sec latency 39.148 ms 2024-03-24T11:54:28.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 201519 10.67 MB/sec execute 334 sec latency 33.120 ms 2024-03-24T11:54:28.812 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:28 smithi032 ceph-mon[66455]: pgmap v89: 97 pgs: 97 active+clean; 325 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 680 op/s 2024-03-24T11:54:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:28 smithi146 ceph-mon[53008]: pgmap v89: 97 pgs: 97 active+clean; 325 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 680 op/s 2024-03-24T11:54:29.456 INFO:teuthology.orchestra.run.smithi032.stdout: 5 201880 10.68 MB/sec execute 335 sec latency 44.630 ms 2024-03-24T11:54:29.879 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:29 smithi032 systemd[1]: Stopping Ceph osd.1 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:54:30.129 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:30 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[39236]: 2024-03-24T11:54:30.041+0000 7f3f71bbe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:54:30.129 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:30 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[39236]: 2024-03-24T11:54:30.041+0000 7f3f71bbe700 -1 osd.1 82 *** Got signal Terminated *** 2024-03-24T11:54:30.129 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:30 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[39236]: 2024-03-24T11:54:30.041+0000 7f3f71bbe700 -1 osd.1 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:54:30.455 INFO:teuthology.orchestra.run.smithi032.stdout: 5 202320 10.68 MB/sec execute 336 sec latency 76.103 ms 2024-03-24T11:54:30.889 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:30 smithi032 ceph-mon[66455]: pgmap v90: 97 pgs: 97 active+clean; 323 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-03-24T11:54:30.889 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:30 smithi032 ceph-mon[66455]: osd.1 marked itself down and dead 2024-03-24T11:54:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:30 smithi146 ceph-mon[53008]: pgmap v90: 97 pgs: 97 active+clean; 323 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-03-24T11:54:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:30 smithi146 ceph-mon[53008]: osd.1 marked itself down and dead 2024-03-24T11:54:31.409 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:31 smithi032 podman[74730]: 2024-03-24 11:54:31.283078527 +0000 UTC m=+1.374695332 container died adc99f43ac4863c5edf4727258d88ca8dc6289a49729ba2643686bf5655a0086 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, vcs-type=git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64) 2024-03-24T11:54:31.457 INFO:teuthology.orchestra.run.smithi032.stdout: 5 202576 10.68 MB/sec execute 337 sec latency 753.459 ms 2024-03-24T11:54:31.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:31 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:54:31.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:31 smithi032 ceph-mon[66455]: failed to encode map e83 with expected crc 2024-03-24T11:54:31.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:31 smithi032 ceph-mon[66455]: osdmap e83: 8 total, 7 up, 8 in 2024-03-24T11:54:31.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:31 smithi032 ceph-mon[66455]: failed to encode map e83 with expected crc 2024-03-24T11:54:31.952 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:31 smithi032 podman[74730]: 2024-03-24 11:54:31.900197034 +0000 UTC m=+1.991813834 container cleanup adc99f43ac4863c5edf4727258d88ca8dc6289a49729ba2643686bf5655a0086 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, distribution-scope=public, name=centos-stream, release=754, version=8, vcs-type=git, GIT_CLEAN=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-24T11:54:31.953 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:31 smithi032 bash[74730]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:54:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:31 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:54:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:31 smithi146 ceph-mon[53008]: failed to encode map e83 with expected crc 2024-03-24T11:54:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:31 smithi146 ceph-mon[53008]: osdmap e83: 8 total, 7 up, 8 in 2024-03-24T11:54:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:31 smithi146 ceph-mon[53008]: failed to encode map e83 with expected crc 2024-03-24T11:54:32.456 INFO:teuthology.orchestra.run.smithi032.stdout: 5 202812 10.67 MB/sec execute 338 sec latency 1030.201 ms 2024-03-24T11:54:32.465 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:32 smithi032 podman[74744]: 2024-03-24 11:54:32.246124399 +0000 UTC m=+0.966987775 container remove adc99f43ac4863c5edf4727258d88ca8dc6289a49729ba2643686bf5655a0086 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., distribution-scope=public, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-24T11:54:32.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: pgmap v92: 97 pgs: 23 peering, 7 stale+active+clean, 67 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 517 op/s 2024-03-24T11:54:32.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: osdmap e84: 8 total, 7 up, 8 in 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:32 smithi032 ceph-mon[66455]: failed to encode map e84 with expected crc 2024-03-24T11:54:32.969 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:32 smithi032 podman[74832]: 2024-03-24 11:54:32.786064927 +0000 UTC m=+0.037531094 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:54:32.969 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:32 smithi032 podman[74832]: 2024-03-24 11:54:32.920996551 +0000 UTC m=+0.172462713 container create 7e32ebb7a7cf8daa32d15d4cc7135f23c33fb9a9e140b372dd50b5d3eb086b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:54:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: pgmap v92: 97 pgs: 23 peering, 7 stale+active+clean, 67 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 517 op/s 2024-03-24T11:54:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: osdmap e84: 8 total, 7 up, 8 in 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:32 smithi146 ceph-mon[53008]: failed to encode map e84 with expected crc 2024-03-24T11:54:33.456 INFO:teuthology.orchestra.run.smithi032.stdout: 5 203162 10.66 MB/sec execute 339 sec latency 70.878 ms 2024-03-24T11:54:33.471 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:33 smithi032 podman[74832]: 2024-03-24 11:54:33.283711674 +0000 UTC m=+0.535177841 container init 7e32ebb7a7cf8daa32d15d4cc7135f23c33fb9a9e140b372dd50b5d3eb086b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:54:33.471 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:33 smithi032 podman[74832]: 2024-03-24 11:54:33.288898069 +0000 UTC m=+0.540364236 container start 7e32ebb7a7cf8daa32d15d4cc7135f23c33fb9a9e140b372dd50b5d3eb086b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T11:54:33.471 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:33 smithi032 podman[74832]: 2024-03-24 11:54:33.452258868 +0000 UTC m=+0.703725042 container attach 7e32ebb7a7cf8daa32d15d4cc7135f23c33fb9a9e140b372dd50b5d3eb086b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T11:54:33.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:33 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 16/1698 objects degraded (0.942%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:33.977 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:33 smithi032 podman[74832]: 2024-03-24 11:54:33.749687365 +0000 UTC m=+1.001153533 container died 7e32ebb7a7cf8daa32d15d4cc7135f23c33fb9a9e140b372dd50b5d3eb086b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T11:54:34.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:33 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 16/1698 objects degraded (0.942%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: 5 203593 10.64 MB/sec execute 340 sec latency 38.667 ms 2024-03-24T11:54:34.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:34 smithi032 ceph-mon[66455]: pgmap v94: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 35 peering, 58 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 746 op/s; 16/1698 objects degraded (0.942%) 2024-03-24T11:54:34.993 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:34 smithi032 podman[74857]: 2024-03-24 11:54:34.821098292 +0000 UTC m=+1.064567537 container remove 7e32ebb7a7cf8daa32d15d4cc7135f23c33fb9a9e140b372dd50b5d3eb086b59 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:54:34.993 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:34 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-24T11:54:34.994 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:34 smithi032 systemd[1]: Stopped Ceph osd.1 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:54:34.994 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:34 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.1.service: Consumed 3min 39.796s CPU time. 2024-03-24T11:54:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:34 smithi146 ceph-mon[53008]: pgmap v94: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 35 peering, 58 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 746 op/s; 16/1698 objects degraded (0.942%) 2024-03-24T11:54:35.244 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:35 smithi032 systemd[1]: Starting Ceph osd.1 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:54:35.457 INFO:teuthology.orchestra.run.smithi032.stdout: 5 203935 10.64 MB/sec execute 341 sec latency 31.727 ms 2024-03-24T11:54:35.714 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:35 smithi032 podman[74957]: 2024-03-24 11:54:35.406347661 +0000 UTC m=+0.025988081 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:54:35.714 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:35 smithi032 podman[74957]: 2024-03-24 11:54:35.638598032 +0000 UTC m=+0.258238452 container create 0082354ddd08b742f02c601469d6976738c7538121b2a8f810026783b9648ef3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:54:36.282 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 podman[74957]: 2024-03-24 11:54:36.210226317 +0000 UTC m=+0.829866728 container init 0082354ddd08b742f02c601469d6976738c7538121b2a8f810026783b9648ef3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, io.buildah.version=1.33.5) 2024-03-24T11:54:36.283 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 podman[74957]: 2024-03-24 11:54:36.215806601 +0000 UTC m=+0.835447019 container start 0082354ddd08b742f02c601469d6976738c7538121b2a8f810026783b9648ef3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:54:36.457 INFO:teuthology.orchestra.run.smithi032.stdout: 5 204420 10.66 MB/sec execute 342 sec latency 46.572 ms 2024-03-24T11:54:36.534 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 podman[74957]: 2024-03-24 11:54:36.379737583 +0000 UTC m=+0.999377999 container attach 0082354ddd08b742f02c601469d6976738c7538121b2a8f810026783b9648ef3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T11:54:36.785 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-mon[66455]: pgmap v95: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 35 peering, 58 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.8 MiB/s wr, 419 op/s; 16/1698 objects degraded (0.942%) 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate[74971]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate[74971]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 bash[74957]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 bash[74957]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 bash[74957]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 bash[74957]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 bash[74957]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 bash[74957]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:54:37.053 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 bash[74957]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-24T11:54:37.054 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate[74971]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:54:37.054 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate[74971]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:54:37.054 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate[74971]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-24T11:54:37.054 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate[74971]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:54:37.054 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate[74971]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-24T11:54:37.054 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:36 smithi032 podman[74957]: 2024-03-24 11:54:36.890898314 +0000 UTC m=+1.510538737 container died 0082354ddd08b742f02c601469d6976738c7538121b2a8f810026783b9648ef3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:54:37.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:36 smithi146 ceph-mon[53008]: pgmap v95: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 35 peering, 58 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.8 MiB/s wr, 419 op/s; 16/1698 objects degraded (0.942%) 2024-03-24T11:54:37.457 INFO:teuthology.orchestra.run.smithi032.stdout: 5 204919 10.66 MB/sec execute 343 sec latency 43.140 ms 2024-03-24T11:54:37.809 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:37 smithi032 ceph-mon[66455]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T11:54:37.810 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:37 smithi032 podman[75102]: 2024-03-24 11:54:37.776797511 +0000 UTC m=+0.879279188 container remove 0082354ddd08b742f02c601469d6976738c7538121b2a8f810026783b9648ef3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:54:38.064 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:38 smithi032 podman[75156]: 2024-03-24 11:54:37.928688042 +0000 UTC m=+0.027796982 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:54:38.064 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:38 smithi032 podman[75156]: 2024-03-24 11:54:38.052963506 +0000 UTC m=+0.152072430 container create 6e2b66b3ab3a8bae53ce7404cf562f1bf3547ef678115d6ccbcbe998cac297de (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:54:38.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:37 smithi146 ceph-mon[53008]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T11:54:38.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 205230 10.65 MB/sec execute 344 sec latency 43.871 ms 2024-03-24T11:54:38.572 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:38 smithi032 podman[75156]: 2024-03-24 11:54:38.443111183 +0000 UTC m=+0.542220125 container init 6e2b66b3ab3a8bae53ce7404cf562f1bf3547ef678115d6ccbcbe998cac297de (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:54:38.572 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:38 smithi032 podman[75156]: 2024-03-24 11:54:38.44908176 +0000 UTC m=+0.548190689 container start 6e2b66b3ab3a8bae53ce7404cf562f1bf3547ef678115d6ccbcbe998cac297de (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T11:54:38.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:38 smithi032 ceph-osd[75174]: -- 172.21.15.32:0/1462936124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f6124ad1e0 con 0x55f61247f000 2024-03-24T11:54:38.823 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:38 smithi032 bash[75156]: 6e2b66b3ab3a8bae53ce7404cf562f1bf3547ef678115d6ccbcbe998cac297de 2024-03-24T11:54:38.823 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:38 smithi032 systemd[1]: Started Ceph osd.1 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:54:38.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:38 smithi032 ceph-mon[66455]: pgmap v96: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 730 op/s; 174/1599 objects degraded (10.882%) 2024-03-24T11:54:38.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:38 smithi032 ceph-mon[66455]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-24T11:54:38.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:38 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:38.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:38 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:38.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:38 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:39.075 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:39 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[75170]: 2024-03-24T11:54:39.030+0000 7f0ff07267c0 -1 Falling back to public interface 2024-03-24T11:54:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:38 smithi146 ceph-mon[53008]: pgmap v96: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 730 op/s; 174/1599 objects degraded (10.882%) 2024-03-24T11:54:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:38 smithi146 ceph-mon[53008]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-24T11:54:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:38 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:38 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:38 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:39.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 205563 10.65 MB/sec execute 345 sec latency 34.649 ms 2024-03-24T11:54:39.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:39 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 175/1593 objects degraded (10.986%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:40.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:39 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 175/1593 objects degraded (10.986%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:40.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 206007 10.66 MB/sec execute 346 sec latency 49.449 ms 2024-03-24T11:54:40.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:40 smithi032 ceph-mon[66455]: pgmap v97: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 681 op/s; 175/1593 objects degraded (10.986%) 2024-03-24T11:54:40.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:40.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:40 smithi146 ceph-mon[53008]: pgmap v97: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 681 op/s; 175/1593 objects degraded (10.986%) 2024-03-24T11:54:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 206296 10.67 MB/sec execute 347 sec latency 48.383 ms 2024-03-24T11:54:42.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 206478 10.65 MB/sec execute 348 sec latency 100.386 ms 2024-03-24T11:54:42.925 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:42 smithi032 ceph-mon[66455]: pgmap v98: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.0 MiB/s wr, 634 op/s; 169/1563 objects degraded (10.813%) 2024-03-24T11:54:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:42 smithi146 ceph-mon[53008]: pgmap v98: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.0 MiB/s wr, 634 op/s; 169/1563 objects degraded (10.813%) 2024-03-24T11:54:43.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 206726 10.63 MB/sec execute 349 sec latency 83.151 ms 2024-03-24T11:54:43.686 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:54:44.202 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:44 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[75170]: 2024-03-24T11:54:44.091+0000 7f0ff07267c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-24T11:54:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 206978 10.60 MB/sec execute 350 sec latency 59.808 ms 2024-03-24T11:54:44.627 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:54:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 45s ago 14m 22.5M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:54:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (93s) 45s ago 14m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:54:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (87s) 82s ago 13m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:54:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (4m) 45s ago 13m 54.5M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:54:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (8m) 45s ago 8m 40.5M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:54:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (8m) 82s ago 8m 16.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:54:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (4m) 45s ago 15m 502M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (3m) 82s ago 13m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (3m) 45s ago 15m 45.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (2m) 82s ago 13m 33.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (8m) 45s ago 8m 253M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (7m) 82s ago 7m 88.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (4m) 45s ago 13m 16.2M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (4m) 82s ago 13m 20.1M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (55s) 45s ago 12m 15.1M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 starting - - - 1937M 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 45s ago 11m 447M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (11m) 45s ago 11m 585M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (11m) 82s ago 11m 566M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:54:44.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (10m) 82s ago 10m 397M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:54:44.630 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (10m) 82s ago 10m 575M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:54:44.630 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (9m) 82s ago 9m 498M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:54:44.630 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (3m) 45s ago 13m 66.6M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:54:44.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:44 smithi032 ceph-mon[66455]: pgmap v99: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 329 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 484 op/s; 176/1590 objects degraded (11.069%) 2024-03-24T11:54:44.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:44 smithi032 ceph-mon[66455]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:44.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:44 smithi032 ceph-mon[66455]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:45.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:44 smithi146 ceph-mon[53008]: pgmap v99: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 329 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 484 op/s; 176/1590 objects degraded (11.069%) 2024-03-24T11:54:45.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:44 smithi146 ceph-mon[53008]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:45.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:44 smithi146 ceph-mon[53008]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:45.328 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:54:45.328 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:54:45.328 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:54:45.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:54:45.329 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:54:45.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:54:45.330 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:54:45.330 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-24T11:54:45.330 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T11:54:45.330 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:54:45.330 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:54:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 207200 10.58 MB/sec execute 351 sec latency 51.947 ms 2024-03-24T11:54:45.499 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:45 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[75170]: 2024-03-24T11:54:45.290+0000 7f0ff07267c0 -1 osd.1 82 log_to_monitors true 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:54:45.821 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "7/23 daemons upgraded", 2024-03-24T11:54:45.822 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:54:45.822 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:54:45.822 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:54:46.016 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:45 smithi032 ceph-mon[66455]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:46.016 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:45 smithi032 ceph-mon[66455]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:54:46.016 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:45 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2335660354' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:46.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:45 smithi146 ceph-mon[53008]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:46.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:45 smithi146 ceph-mon[53008]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:54:46.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:45 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/2335660354' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:46.266 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[75170]: 2024-03-24T11:54:46.204+0000 7f0fddb76700 -1 osd.1 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 176/1590 objects degraded (11.069%), 24 pgs degraded 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout: osd.1 (root=default,host=smithi032) is down 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 176/1590 objects degraded (11.069%), 24 pgs degraded 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-03-24T11:54:46.442 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-03-24T11:54:46.443 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-03-24T11:54:46.444 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-03-24T11:54:46.444 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-03-24T11:54:46.444 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-03-24T11:54:46.444 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-03-24T11:54:46.444 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.a is active+undersized+degraded, acting [6,7] 2024-03-24T11:54:46.444 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-03-24T11:54:46.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 207423 10.58 MB/sec execute 352 sec latency 65.063 ms 2024-03-24T11:54:47.053 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: pgmap v100: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 329 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.7 MiB/s wr, 467 op/s; 176/1590 objects degraded (11.069%) 2024-03-24T11:54:47.053 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:47.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:54:47.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: failed to encode map e85 with expected crc 2024-03-24T11:54:47.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: osdmap e85: 8 total, 7 up, 8 in 2024-03-24T11:54:47.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:54:47.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: failed to encode map e85 with expected crc 2024-03-24T11:54:47.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' 2024-03-24T11:54:47.054 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:46 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3477072764' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:54:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: pgmap v100: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 329 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.7 MiB/s wr, 467 op/s; 176/1590 objects degraded (11.069%) 2024-03-24T11:54:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:54:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:54:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: failed to encode map e85 with expected crc 2024-03-24T11:54:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: osdmap e85: 8 total, 7 up, 8 in 2024-03-24T11:54:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:54:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: failed to encode map e85 with expected crc 2024-03-24T11:54:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: from='osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752]' entity='osd.1' 2024-03-24T11:54:47.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:46 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3477072764' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:54:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 207740 10.59 MB/sec execute 353 sec latency 295.098 ms 2024-03-24T11:54:48.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:54:48.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: failed to encode map e86 with expected crc 2024-03-24T11:54:48.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752] boot 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: failed to encode map e86 with expected crc 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 176/1590 objects degraded (11.069%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:47 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:54:48.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: failed to encode map e86 with expected crc 2024-03-24T11:54:48.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: osd.1 [v2:172.21.15.32:6810/30310752,v1:172.21.15.32:6811/30310752] boot 2024-03-24T11:54:48.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:54:48.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:54:48.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: failed to encode map e86 with expected crc 2024-03-24T11:54:48.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 176/1590 objects degraded (11.069%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:48.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:54:48.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:47 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:48.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 207740 10.56 MB/sec execute 354 sec latency 1295.148 ms 2024-03-24T11:54:49.166 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: pgmap v104: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 4.2 MiB/s wr, 368 op/s; 175/1557 objects degraded (11.240%) 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e88 with expected crc 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e88 with expected crc 2024-03-24T11:54:49.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:49 smithi032 ceph-mon[66455]: failed to encode map e88 with expected crc 2024-03-24T11:54:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e87 with expected crc 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: pgmap v104: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 4.2 MiB/s wr, 368 op/s; 175/1557 objects degraded (11.240%) 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e88 with expected crc 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e88 with expected crc 2024-03-24T11:54:49.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:49 smithi146 ceph-mon[53008]: failed to encode map e88 with expected crc 2024-03-24T11:54:49.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 207908 10.53 MB/sec execute 355 sec latency 1570.355 ms 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: failed to encode map e89 with expected crc 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:50.373 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:50.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: failed to encode map e88 with expected crc 2024-03-24T11:54:50.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:54:50.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: failed to encode map e89 with expected crc 2024-03-24T11:54:50.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:50.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:50.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 207939 10.51 MB/sec execute 356 sec latency 970.980 ms 2024-03-24T11:54:51.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:51 smithi032 ceph-mon[66455]: failed to encode map e89 with expected crc 2024-03-24T11:54:51.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:51 smithi032 ceph-mon[66455]: failed to encode map e89 with expected crc 2024-03-24T11:54:51.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:51 smithi032 ceph-mon[66455]: failed to encode map e89 with expected crc 2024-03-24T11:54:51.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:51 smithi032 ceph-mon[66455]: pgmap v107: 97 pgs: 1 remapped+peering, 13 active+undersized, 23 active+undersized+degraded, 60 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 491 op/s; 159/1527 objects degraded (10.413%) 2024-03-24T11:54:51.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:51 smithi032 ceph-mon[66455]: failed to encode map e90 with expected crc 2024-03-24T11:54:51.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:51 smithi032 ceph-mon[66455]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:54:51.127 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:51 smithi032 ceph-mon[66455]: failed to encode map e90 with expected crc 2024-03-24T11:54:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:51 smithi146 ceph-mon[53008]: failed to encode map e89 with expected crc 2024-03-24T11:54:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:51 smithi146 ceph-mon[53008]: failed to encode map e89 with expected crc 2024-03-24T11:54:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:51 smithi146 ceph-mon[53008]: failed to encode map e89 with expected crc 2024-03-24T11:54:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:51 smithi146 ceph-mon[53008]: pgmap v107: 97 pgs: 1 remapped+peering, 13 active+undersized, 23 active+undersized+degraded, 60 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 491 op/s; 159/1527 objects degraded (10.413%) 2024-03-24T11:54:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:51 smithi146 ceph-mon[53008]: failed to encode map e90 with expected crc 2024-03-24T11:54:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:51 smithi146 ceph-mon[53008]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:54:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:51 smithi146 ceph-mon[53008]: failed to encode map e90 with expected crc 2024-03-24T11:54:51.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 208299 10.50 MB/sec execute 357 sec latency 1028.567 ms 2024-03-24T11:54:52.154 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:52 smithi032 ceph-mon[66455]: failed to encode map e90 with expected crc 2024-03-24T11:54:52.154 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:52 smithi032 ceph-mon[66455]: failed to encode map e90 with expected crc 2024-03-24T11:54:52.155 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:52 smithi032 ceph-mon[66455]: failed to encode map e90 with expected crc 2024-03-24T11:54:52.155 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:52 smithi032 ceph-mon[66455]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:54:52.155 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:52 smithi032 ceph-mon[66455]: failed to encode map e91 with expected crc 2024-03-24T11:54:52.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:52 smithi146 ceph-mon[53008]: failed to encode map e90 with expected crc 2024-03-24T11:54:52.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:52 smithi146 ceph-mon[53008]: failed to encode map e90 with expected crc 2024-03-24T11:54:52.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:52 smithi146 ceph-mon[53008]: failed to encode map e90 with expected crc 2024-03-24T11:54:52.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:52 smithi146 ceph-mon[53008]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:54:52.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:52 smithi146 ceph-mon[53008]: failed to encode map e91 with expected crc 2024-03-24T11:54:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 208476 10.48 MB/sec execute 358 sec latency 901.898 ms 2024-03-24T11:54:53.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: pgmap v110: 97 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 2 active+undersized, 1 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 87 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s wr, 209 op/s; 44/1569 objects degraded (2.804%); 541 KiB/s, 48 keys/s, 29 objects/s recovering 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 159/1527 objects degraded (10.413%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e92 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e92 with expected crc 2024-03-24T11:54:53.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:53 smithi032 ceph-mon[66455]: failed to encode map e92 with expected crc 2024-03-24T11:54:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e91 with expected crc 2024-03-24T11:54:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: pgmap v110: 97 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 2 active+undersized, 1 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 87 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s wr, 209 op/s; 44/1569 objects degraded (2.804%); 541 KiB/s, 48 keys/s, 29 objects/s recovering 2024-03-24T11:54:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 159/1527 objects degraded (10.413%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:54:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e92 with expected crc 2024-03-24T11:54:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e92 with expected crc 2024-03-24T11:54:53.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:53 smithi146 ceph-mon[53008]: failed to encode map e92 with expected crc 2024-03-24T11:54:53.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 208771 10.47 MB/sec execute 359 sec latency 950.913 ms 2024-03-24T11:54:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:54 smithi032 ceph-mon[66455]: failed to encode map e92 with expected crc 2024-03-24T11:54:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:54 smithi032 ceph-mon[66455]: failed to encode map e93 with expected crc 2024-03-24T11:54:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:54 smithi032 ceph-mon[66455]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:54:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:54 smithi032 ceph-mon[66455]: failed to encode map e93 with expected crc 2024-03-24T11:54:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:54 smithi032 ceph-mon[66455]: failed to encode map e93 with expected crc 2024-03-24T11:54:54.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:54 smithi146 ceph-mon[53008]: failed to encode map e92 with expected crc 2024-03-24T11:54:54.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:54 smithi146 ceph-mon[53008]: failed to encode map e93 with expected crc 2024-03-24T11:54:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:54 smithi146 ceph-mon[53008]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:54:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:54 smithi146 ceph-mon[53008]: failed to encode map e93 with expected crc 2024-03-24T11:54:54.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:54 smithi146 ceph-mon[53008]: failed to encode map e93 with expected crc 2024-03-24T11:54:54.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 209132 10.47 MB/sec execute 360 sec latency 974.984 ms 2024-03-24T11:54:55.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 209485 10.47 MB/sec execute 361 sec latency 1022.037 ms 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: failed to encode map e93 with expected crc 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: pgmap v113: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 93 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.1 MiB/s wr, 221 op/s; 40/1569 objects degraded (2.549%); 578 KiB/s, 48 keys/s, 41 objects/s recovering 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: failed to encode map e94 with expected crc 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:54:55.486 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: failed to encode map e94 with expected crc 2024-03-24T11:54:55.487 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: failed to encode map e94 with expected crc 2024-03-24T11:54:55.487 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:55.487 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:55.487 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:55.487 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:55.487 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:54:55.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: failed to encode map e93 with expected crc 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: pgmap v113: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 93 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.1 MiB/s wr, 221 op/s; 40/1569 objects degraded (2.549%); 578 KiB/s, 48 keys/s, 41 objects/s recovering 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: failed to encode map e94 with expected crc 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: failed to encode map e94 with expected crc 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: failed to encode map e94 with expected crc 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:55.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:55.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:55.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:54:55.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:54:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 209838 10.48 MB/sec execute 362 sec latency 72.449 ms 2024-03-24T11:54:56.517 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:56 smithi032 ceph-mon[66455]: failed to encode map e94 with expected crc 2024-03-24T11:54:56.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:56 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:54:56.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:56 smithi032 ceph-mon[66455]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:54:56.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:56.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:56.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:56 smithi146 ceph-mon[53008]: failed to encode map e94 with expected crc 2024-03-24T11:54:56.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:56 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:54:56.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:56 smithi146 ceph-mon[53008]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:54:56.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:54:56.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:57.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 210164 10.48 MB/sec execute 363 sec latency 46.152 ms 2024-03-24T11:54:57.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:57 smithi032 ceph-mon[66455]: pgmap v115: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 93 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 1.5 MiB/s wr, 94 op/s; 40/1569 objects degraded (2.549%); 36 KiB/s, 32 objects/s recovering 2024-03-24T11:54:57.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:57 smithi146 ceph-mon[53008]: pgmap v115: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 93 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 1.5 MiB/s wr, 94 op/s; 40/1569 objects degraded (2.549%); 36 KiB/s, 32 objects/s recovering 2024-03-24T11:54:58.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 210419 10.46 MB/sec execute 364 sec latency 38.375 ms 2024-03-24T11:54:58.549 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:58 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 40/1569 objects degraded (2.549%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:58.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:58 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 40/1569 objects degraded (2.549%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:54:59.458 INFO:teuthology.orchestra.run.smithi032.stdout: 5 210690 10.45 MB/sec execute 365 sec latency 50.065 ms 2024-03-24T11:54:59.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:59 smithi032 ceph-mon[66455]: pgmap v116: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.3 MiB/s wr, 426 op/s; 25 KiB/s, 18 objects/s recovering 2024-03-24T11:54:59.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:59 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/1569 objects degraded (2.549%), 4 pgs degraded) 2024-03-24T11:54:59.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:54:59 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:54:59.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:59 smithi146 ceph-mon[53008]: pgmap v116: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.3 MiB/s wr, 426 op/s; 25 KiB/s, 18 objects/s recovering 2024-03-24T11:54:59.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:59 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/1569 objects degraded (2.549%), 4 pgs degraded) 2024-03-24T11:54:59.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:54:59 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:55:00.459 INFO:teuthology.orchestra.run.smithi032.stdout: 5 210973 10.43 MB/sec execute 366 sec latency 43.661 ms 2024-03-24T11:55:01.459 INFO:teuthology.orchestra.run.smithi032.stdout: 5 211247 10.43 MB/sec execute 367 sec latency 50.142 ms 2024-03-24T11:55:01.648 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:01 smithi032 ceph-mon[66455]: pgmap v117: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.9 MiB/s wr, 367 op/s; 21 KiB/s, 12 objects/s recovering 2024-03-24T11:55:01.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:01 smithi146 ceph-mon[53008]: pgmap v117: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.9 MiB/s wr, 367 op/s; 21 KiB/s, 12 objects/s recovering 2024-03-24T11:55:02.459 INFO:teuthology.orchestra.run.smithi032.stdout: 5 211580 10.43 MB/sec execute 368 sec latency 51.619 ms 2024-03-24T11:55:03.459 INFO:teuthology.orchestra.run.smithi032.stdout: 5 211974 10.43 MB/sec execute 369 sec latency 85.522 ms 2024-03-24T11:55:03.696 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:03 smithi032 ceph-mon[66455]: pgmap v118: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.1 MiB/s wr, 533 op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:55:03.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:03 smithi146 ceph-mon[53008]: pgmap v118: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.1 MiB/s wr, 533 op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:55:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: 5 212343 10.41 MB/sec execute 370 sec latency 81.136 ms 2024-03-24T11:55:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: 5 212693 10.42 MB/sec execute 371 sec latency 37.979 ms 2024-03-24T11:55:05.538 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:05 smithi032 ceph-mon[66455]: pgmap v119: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 471 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:55:05.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:05 smithi146 ceph-mon[53008]: pgmap v119: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 471 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:55:06.459 INFO:teuthology.orchestra.run.smithi032.stdout: 5 213123 10.43 MB/sec execute 372 sec latency 49.920 ms 2024-03-24T11:55:06.898 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:06 smithi032 ceph-mon[66455]: pgmap v120: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 422 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:55:06.898 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:06 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:06 smithi146 ceph-mon[53008]: pgmap v120: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 422 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:55:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:06 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 213629 10.45 MB/sec execute 373 sec latency 32.908 ms 2024-03-24T11:55:08.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 214023 10.43 MB/sec execute 374 sec latency 43.900 ms 2024-03-24T11:55:08.916 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:08 smithi032 ceph-mon[66455]: pgmap v121: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.9 MiB/s wr, 578 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:55:09.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:08 smithi146 ceph-mon[53008]: pgmap v121: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.9 MiB/s wr, 578 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:55:09.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 214390 10.43 MB/sec execute 375 sec latency 29.403 ms 2024-03-24T11:55:09.926 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:09 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:55:10.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:09 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:55:10.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 214728 10.43 MB/sec execute 376 sec latency 32.460 ms 2024-03-24T11:55:10.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:10 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:55:10.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:10 smithi032 ceph-mon[66455]: Upgrade: osd.2 is safe to restart 2024-03-24T11:55:10.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:10 smithi032 ceph-mon[66455]: pgmap v122: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.3 MiB/s wr, 395 op/s 2024-03-24T11:55:10.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:10.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:10 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:55:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:10 smithi146 ceph-mon[53008]: Upgrade: osd.2 is safe to restart 2024-03-24T11:55:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:10 smithi146 ceph-mon[53008]: pgmap v122: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.3 MiB/s wr, 395 op/s 2024-03-24T11:55:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 215087 10.44 MB/sec execute 377 sec latency 44.413 ms 2024-03-24T11:55:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 215513 10.43 MB/sec execute 378 sec latency 41.240 ms 2024-03-24T11:55:13.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:13 smithi032 ceph-mon[66455]: pgmap v123: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.5 MiB/s wr, 531 op/s 2024-03-24T11:55:13.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:13 smithi032 ceph-mon[66455]: Upgrade: Updating osd.2 2024-03-24T11:55:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:13 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:13 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:55:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:13 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:13 smithi032 ceph-mon[66455]: Deploying daemon osd.2 on smithi032 2024-03-24T11:55:13.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 215904 10.42 MB/sec execute 379 sec latency 41.233 ms 2024-03-24T11:55:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:13 smithi146 ceph-mon[53008]: pgmap v123: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.5 MiB/s wr, 531 op/s 2024-03-24T11:55:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:13 smithi146 ceph-mon[53008]: Upgrade: Updating osd.2 2024-03-24T11:55:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:13 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:13 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:55:13.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:13 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:13.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:13 smithi146 ceph-mon[53008]: Deploying daemon osd.2 on smithi032 2024-03-24T11:55:14.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 216214 10.41 MB/sec execute 380 sec latency 47.230 ms 2024-03-24T11:55:15.270 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:15 smithi032 ceph-mon[66455]: pgmap v124: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-03-24T11:55:15.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 216685 10.43 MB/sec execute 381 sec latency 35.148 ms 2024-03-24T11:55:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:15 smithi146 ceph-mon[53008]: pgmap v124: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-03-24T11:55:16.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 217215 10.45 MB/sec execute 382 sec latency 48.033 ms 2024-03-24T11:55:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:55:17.303 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:17 smithi032 ceph-mon[66455]: pgmap v125: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 470 op/s 2024-03-24T11:55:17.436 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:17 smithi146 ceph-mon[53008]: pgmap v125: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 470 op/s 2024-03-24T11:55:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 217511 10.44 MB/sec execute 383 sec latency 46.275 ms 2024-03-24T11:55:17.554 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:17 smithi032 systemd[1]: Stopping Ceph osd.2 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:55:17.808 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:17 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[42937]: 2024-03-24T11:55:17.759+0000 7fe72a9b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:55:17.809 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:17 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[42937]: 2024-03-24T11:55:17.759+0000 7fe72a9b9700 -1 osd.2 94 *** Got signal Terminated *** 2024-03-24T11:55:17.809 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:17 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[42937]: 2024-03-24T11:55:17.759+0000 7fe72a9b9700 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:55:17.969 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 29s ago 14m 22.5M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (2m) 29s ago 14m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (2m) 116s ago 14m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 29s ago 14m 52.2M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (8m) 29s ago 8m 40.9M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (8m) 116s ago 8m 16.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (4m) 29s ago 16m 503M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (3m) 116s ago 14m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (3m) 29s ago 16m 48.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (3m) 116s ago 14m 33.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:55:17.970 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (8m) 29s ago 8m 255M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (8m) 116s ago 8m 88.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (5m) 29s ago 14m 18.7M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (4m) 116s ago 13m 20.1M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (88s) 29s ago 13m 256M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (40s) 29s ago 12m 12.2M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 29s ago 12m 490M 1937M 17.2.0 e1d6a67b021e 22f004d55e3b 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (11m) 29s ago 11m 368M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (11m) 116s ago 11m 566M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (11m) 116s ago 11m 397M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (10m) 116s ago 10m 575M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (10m) 116s ago 10m 498M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:55:17.971 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (4m) 29s ago 13m 66.6M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:55:18.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:18 smithi032 ceph-mon[66455]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:18.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:18 smithi032 ceph-mon[66455]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:18.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:18 smithi032 ceph-mon[66455]: osd.2 marked itself down and dead 2024-03-24T11:55:18.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 217859 10.43 MB/sec execute 384 sec latency 234.931 ms 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:18.523 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 6 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:55:18.524 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:55:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:18 smithi146 ceph-mon[53008]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:18 smithi146 ceph-mon[53008]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:18 smithi146 ceph-mon[53008]: osd.2 marked itself down and dead 2024-03-24T11:55:18.981 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:55:18.981 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:55:18.981 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:55:18.982 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:55:18.982 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:55:18.982 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:55:18.982 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:55:18.982 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:55:18.982 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:55:18.983 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "8/23 daemons upgraded", 2024-03-24T11:55:18.983 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:55:18.983 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:55:18.983 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:55:19.162 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:18 smithi032 podman[77346]: 2024-03-24 11:55:18.849359262 +0000 UTC m=+1.304097689 container died 22f004d55e3bbd45f86ffd9f7145bf929c8f22982113cb42f8f169584ef91338 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_CLEAN=True, ceph=True, release=754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, version=8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-24T11:55:19.412 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:19 smithi032 ceph-mon[66455]: pgmap v126: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 699 op/s 2024-03-24T11:55:19.412 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:19 smithi032 ceph-mon[66455]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:19.412 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:19 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:55:19.412 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:19 smithi032 ceph-mon[66455]: failed to encode map e95 with expected crc 2024-03-24T11:55:19.413 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:19 smithi032 ceph-mon[66455]: osdmap e95: 8 total, 7 up, 8 in 2024-03-24T11:55:19.413 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:19 smithi032 ceph-mon[66455]: failed to encode map e95 with expected crc 2024-03-24T11:55:19.413 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:19 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/4175765441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:19.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 218058 10.41 MB/sec execute 385 sec latency 971.610 ms 2024-03-24T11:55:19.543 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down 2024-03-24T11:55:19.543 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T11:55:19.543 INFO:teuthology.orchestra.run.smithi032.stdout: osd.2 (root=default,host=smithi032) is down 2024-03-24T11:55:19.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:19 smithi146 ceph-mon[53008]: pgmap v126: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 699 op/s 2024-03-24T11:55:19.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:19 smithi146 ceph-mon[53008]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:19.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:19 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:55:19.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:19 smithi146 ceph-mon[53008]: failed to encode map e95 with expected crc 2024-03-24T11:55:19.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:19 smithi146 ceph-mon[53008]: osdmap e95: 8 total, 7 up, 8 in 2024-03-24T11:55:19.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:19 smithi146 ceph-mon[53008]: failed to encode map e95 with expected crc 2024-03-24T11:55:19.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:19 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/4175765441' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:19.668 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:19 smithi032 podman[77346]: 2024-03-24 11:55:19.434063216 +0000 UTC m=+1.888801644 container cleanup 22f004d55e3bbd45f86ffd9f7145bf929c8f22982113cb42f8f169584ef91338 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.component=centos-stream-container, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.k8s.display-name=CentOS Stream 8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T11:55:19.668 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:19 smithi032 bash[77346]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:55:19.930 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:19 smithi032 podman[77427]: 2024-03-24 11:55:19.731214627 +0000 UTC m=+0.877097512 container remove 22f004d55e3bbd45f86ffd9f7145bf929c8f22982113cb42f8f169584ef91338 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, release=754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8) 2024-03-24T11:55:20.190 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:20 smithi032 podman[77535]: 2024-03-24 11:55:20.038603211 +0000 UTC m=+0.026974360 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: osdmap e96: 8 total, 7 up, 8 in 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:20 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3513604063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:55:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 218356 10.41 MB/sec execute 386 sec latency 68.598 ms 2024-03-24T11:55:20.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:20.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: osdmap e96: 8 total, 7 up, 8 in 2024-03-24T11:55:20.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: failed to encode map e96 with expected crc 2024-03-24T11:55:20.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:20 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3513604063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:55:20.698 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:20 smithi032 podman[77535]: 2024-03-24 11:55:20.49998169 +0000 UTC m=+0.488352829 container create e23c9cd596d875e204d6a8d4e7fcf6beb95c9f933bbed4e2df0c7dae7b85375e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:55:21.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:21 smithi032 ceph-mon[66455]: pgmap v129: 97 pgs: 5 stale+active+clean, 92 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 546 op/s 2024-03-24T11:55:21.459 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:21 smithi032 podman[77535]: 2024-03-24 11:55:21.351836661 +0000 UTC m=+1.340207821 container init e23c9cd596d875e204d6a8d4e7fcf6beb95c9f933bbed4e2df0c7dae7b85375e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T11:55:21.459 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:21 smithi032 podman[77535]: 2024-03-24 11:55:21.359331174 +0000 UTC m=+1.347702316 container start e23c9cd596d875e204d6a8d4e7fcf6beb95c9f933bbed4e2df0c7dae7b85375e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T11:55:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 218694 10.41 MB/sec execute 387 sec latency 91.473 ms 2024-03-24T11:55:21.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:21 smithi146 ceph-mon[53008]: pgmap v129: 97 pgs: 5 stale+active+clean, 92 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 546 op/s 2024-03-24T11:55:21.709 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:21 smithi032 podman[77535]: 2024-03-24 11:55:21.532096694 +0000 UTC m=+1.520467838 container attach e23c9cd596d875e204d6a8d4e7fcf6beb95c9f933bbed4e2df0c7dae7b85375e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T11:55:21.960 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:21 smithi032 podman[77535]: 2024-03-24 11:55:21.823676206 +0000 UTC m=+1.812047361 container died e23c9cd596d875e204d6a8d4e7fcf6beb95c9f933bbed4e2df0c7dae7b85375e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T11:55:22.462 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:22 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 111/1572 objects degraded (7.061%), 13 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:22.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 219026 10.40 MB/sec execute 388 sec latency 49.246 ms 2024-03-24T11:55:22.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:22 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 111/1572 objects degraded (7.061%), 13 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:22.973 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:22 smithi032 podman[77563]: 2024-03-24 11:55:22.847216104 +0000 UTC m=+1.009342872 container remove e23c9cd596d875e204d6a8d4e7fcf6beb95c9f933bbed4e2df0c7dae7b85375e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T11:55:22.973 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:22 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-24T11:55:22.973 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:22 smithi032 systemd[1]: Stopped Ceph osd.2 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:55:22.973 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:22 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.2.service: Consumed 2min 47.747s CPU time. 2024-03-24T11:55:23.223 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:23 smithi032 systemd[1]: Starting Ceph osd.2 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:55:23.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 219384 10.39 MB/sec execute 389 sec latency 53.221 ms 2024-03-24T11:55:23.474 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:23 smithi032 ceph-mon[66455]: pgmap v130: 97 pgs: 5 active+undersized, 13 active+undersized+degraded, 2 stale+active+clean, 77 active+clean; 336 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.6 MiB/s wr, 613 op/s; 111/1572 objects degraded (7.061%) 2024-03-24T11:55:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:23 smithi146 ceph-mon[53008]: pgmap v130: 97 pgs: 5 active+undersized, 13 active+undersized+degraded, 2 stale+active+clean, 77 active+clean; 336 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.6 MiB/s wr, 613 op/s; 111/1572 objects degraded (7.061%) 2024-03-24T11:55:23.727 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:23 smithi032 podman[77661]: 2024-03-24 11:55:23.38775739 +0000 UTC m=+0.029126787 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:55:23.728 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:23 smithi032 podman[77661]: 2024-03-24 11:55:23.52335708 +0000 UTC m=+0.164726464 container create caf5ee7d99dc3849e9b512b065ede79811f8619a2ac6b3f0b81f1d50f73f302e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T11:55:23.978 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:23 smithi032 podman[77661]: 2024-03-24 11:55:23.906885202 +0000 UTC m=+0.548254591 container init caf5ee7d99dc3849e9b512b065ede79811f8619a2ac6b3f0b81f1d50f73f302e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T11:55:23.978 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:23 smithi032 podman[77661]: 2024-03-24 11:55:23.912814222 +0000 UTC m=+0.554183631 container start caf5ee7d99dc3849e9b512b065ede79811f8619a2ac6b3f0b81f1d50f73f302e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:55:24.235 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 podman[77661]: 2024-03-24 11:55:24.031391705 +0000 UTC m=+0.672761090 container attach caf5ee7d99dc3849e9b512b065ede79811f8619a2ac6b3f0b81f1d50f73f302e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T11:55:24.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 219633 10.39 MB/sec execute 390 sec latency 72.628 ms 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate[77673]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate[77673]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 bash[77661]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 bash[77661]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 bash[77661]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 bash[77661]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 bash[77661]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 bash[77661]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 bash[77661]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate[77673]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate[77673]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:55:24.755 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate[77673]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-24T11:55:24.756 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate[77673]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:55:24.756 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate[77673]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-24T11:55:24.756 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:24 smithi032 podman[77661]: 2024-03-24 11:55:24.545077592 +0000 UTC m=+1.186446972 container died caf5ee7d99dc3849e9b512b065ede79811f8619a2ac6b3f0b81f1d50f73f302e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T11:55:25.263 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:25 smithi032 ceph-mon[66455]: pgmap v131: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.5 MiB/s wr, 656 op/s; 145/1563 objects degraded (9.277%) 2024-03-24T11:55:25.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 220040 10.39 MB/sec execute 391 sec latency 36.411 ms 2024-03-24T11:55:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:25 smithi146 ceph-mon[53008]: pgmap v131: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.5 MiB/s wr, 656 op/s; 145/1563 objects degraded (9.277%) 2024-03-24T11:55:25.765 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:25 smithi032 podman[77795]: 2024-03-24 11:55:25.603454688 +0000 UTC m=+1.041920804 container remove caf5ee7d99dc3849e9b512b065ede79811f8619a2ac6b3f0b81f1d50f73f302e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:55:26.016 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:25 smithi032 podman[77852]: 2024-03-24 11:55:25.761085948 +0000 UTC m=+0.025316479 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:55:26.279 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:26 smithi032 podman[77852]: 2024-03-24 11:55:26.049159929 +0000 UTC m=+0.313390459 container create ab8a1998f9b73f98d6f4cf61a2976c7c3569848f10a0990164d8ee1e26f9950f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T11:55:26.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 220388 10.40 MB/sec execute 392 sec latency 44.772 ms 2024-03-24T11:55:26.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:26 smithi032 ceph-mon[66455]: pgmap v132: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.3 MiB/s wr, 313 op/s; 145/1563 objects degraded (9.277%) 2024-03-24T11:55:26.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:26.801 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:26.802 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:26 smithi032 podman[77852]: 2024-03-24 11:55:26.560503531 +0000 UTC m=+0.824734123 container init ab8a1998f9b73f98d6f4cf61a2976c7c3569848f10a0990164d8ee1e26f9950f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T11:55:26.802 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:26 smithi032 podman[77852]: 2024-03-24 11:55:26.566254351 +0000 UTC m=+0.830484881 container start ab8a1998f9b73f98d6f4cf61a2976c7c3569848f10a0990164d8ee1e26f9950f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T11:55:26.802 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:26 smithi032 ceph-osd[77871]: -- 172.21.15.32:0/100631938 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b49b6091e0 con 0x55b49b5db000 2024-03-24T11:55:26.802 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:26 smithi032 bash[77852]: ab8a1998f9b73f98d6f4cf61a2976c7c3569848f10a0990164d8ee1e26f9950f 2024-03-24T11:55:26.802 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:26 smithi032 systemd[1]: Started Ceph osd.2 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:55:27.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:26 smithi146 ceph-mon[53008]: pgmap v132: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.3 MiB/s wr, 313 op/s; 145/1563 objects degraded (9.277%) 2024-03-24T11:55:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:27.304 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:27 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[77867]: 2024-03-24T11:55:27.134+0000 7f4e4efbc7c0 -1 Falling back to public interface 2024-03-24T11:55:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 220732 10.40 MB/sec execute 393 sec latency 43.218 ms 2024-03-24T11:55:27.820 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:27 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:27.821 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:27 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:27.821 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:27 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:55:27.821 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:27 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 145/1563 objects degraded (9.277%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:28.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:27 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:28.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:27 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:28.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:27 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:55:28.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:27 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 145/1563 objects degraded (9.277%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:28.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 220959 10.39 MB/sec execute 394 sec latency 62.237 ms 2024-03-24T11:55:28.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:28 smithi032 ceph-mon[66455]: pgmap v133: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 484 op/s; 147/1650 objects degraded (8.909%) 2024-03-24T11:55:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:28 smithi146 ceph-mon[53008]: pgmap v133: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 484 op/s; 147/1650 objects degraded (8.909%) 2024-03-24T11:55:29.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 221154 10.37 MB/sec execute 395 sec latency 72.522 ms 2024-03-24T11:55:30.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 221443 10.35 MB/sec execute 396 sec latency 62.995 ms 2024-03-24T11:55:30.907 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:30 smithi032 ceph-mon[66455]: pgmap v134: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 453 op/s; 148/1653 objects degraded (8.953%) 2024-03-24T11:55:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:30 smithi146 ceph-mon[53008]: pgmap v134: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 453 op/s; 148/1653 objects degraded (8.953%) 2024-03-24T11:55:31.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 221637 10.34 MB/sec execute 397 sec latency 72.582 ms 2024-03-24T11:55:32.274 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:32 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[77867]: 2024-03-24T11:55:32.235+0000 7f4e4efbc7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-24T11:55:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: 5 221938 10.34 MB/sec execute 398 sec latency 49.442 ms 2024-03-24T11:55:32.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:32 smithi032 ceph-mon[66455]: pgmap v135: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.0 MiB/s wr, 532 op/s; 145/1590 objects degraded (9.119%) 2024-03-24T11:55:32.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:32 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 145/1590 objects degraded (9.119%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:32 smithi146 ceph-mon[53008]: pgmap v135: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.0 MiB/s wr, 532 op/s; 145/1590 objects degraded (9.119%) 2024-03-24T11:55:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:32 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 145/1590 objects degraded (9.119%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:33.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 222378 10.35 MB/sec execute 399 sec latency 63.461 ms 2024-03-24T11:55:33.494 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:33 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[77867]: 2024-03-24T11:55:33.431+0000 7f4e4efbc7c0 -1 osd.2 94 log_to_monitors true 2024-03-24T11:55:33.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:33 smithi032 ceph-mon[66455]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:55:34.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:33 smithi146 ceph-mon[53008]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:55:34.247 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 11:55:34 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[77867]: 2024-03-24T11:55:34.195+0000 7f4e3c40c700 -1 osd.2 94 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:55:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 222774 10.34 MB/sec execute 400 sec latency 54.703 ms 2024-03-24T11:55:35.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:34 smithi032 ceph-mon[66455]: pgmap v136: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.2 MiB/s wr, 389 op/s; 146/1587 objects degraded (9.200%) 2024-03-24T11:55:35.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:34 smithi032 ceph-mon[66455]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:55:35.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:34 smithi032 ceph-mon[66455]: failed to encode map e97 with expected crc 2024-03-24T11:55:35.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:34 smithi032 ceph-mon[66455]: osdmap e97: 8 total, 7 up, 8 in 2024-03-24T11:55:35.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:34 smithi032 ceph-mon[66455]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:55:35.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:34 smithi032 ceph-mon[66455]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' 2024-03-24T11:55:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:34 smithi146 ceph-mon[53008]: pgmap v136: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.2 MiB/s wr, 389 op/s; 146/1587 objects degraded (9.200%) 2024-03-24T11:55:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:34 smithi146 ceph-mon[53008]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:55:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:34 smithi146 ceph-mon[53008]: failed to encode map e97 with expected crc 2024-03-24T11:55:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:34 smithi146 ceph-mon[53008]: osdmap e97: 8 total, 7 up, 8 in 2024-03-24T11:55:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:34 smithi146 ceph-mon[53008]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:55:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:34 smithi146 ceph-mon[53008]: from='osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818]' entity='osd.2' 2024-03-24T11:55:35.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 223158 10.34 MB/sec execute 401 sec latency 241.662 ms 2024-03-24T11:55:36.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:35 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:55:36.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:35 smithi146 ceph-mon[53008]: osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818] boot 2024-03-24T11:55:36.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:35 smithi146 ceph-mon[53008]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:55:36.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:55:36.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:35 smithi146 ceph-mon[53008]: failed to encode map e98 with expected crc 2024-03-24T11:55:36.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:36.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:35 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:55:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:35 smithi032 ceph-mon[66455]: osd.2 [v2:172.21.15.32:6818/815071818,v1:172.21.15.32:6819/815071818] boot 2024-03-24T11:55:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:35 smithi032 ceph-mon[66455]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:55:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:55:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:35 smithi032 ceph-mon[66455]: failed to encode map e98 with expected crc 2024-03-24T11:55:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:36.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:36.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 223207 10.32 MB/sec execute 402 sec latency 1241.693 ms 2024-03-24T11:55:37.328 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: pgmap v139: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.6 MiB/s wr, 257 op/s; 146/1587 objects degraded (9.200%) 2024-03-24T11:55:37.328 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.328 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:55:37.328 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.328 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.328 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:37.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:37.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 223207 10.29 MB/sec execute 403 sec latency 2241.728 ms 2024-03-24T11:55:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: pgmap v139: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 72 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.6 MiB/s wr, 257 op/s; 146/1587 objects degraded (9.200%) 2024-03-24T11:55:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:55:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: failed to encode map e99 with expected crc 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:37.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:38.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:55:38.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:38 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 146/1587 objects degraded (9.200%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:38.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 223552 10.30 MB/sec execute 404 sec latency 2249.508 ms 2024-03-24T11:55:38.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:55:38.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: failed to encode map e100 with expected crc 2024-03-24T11:55:38.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:38 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 146/1587 objects degraded (9.200%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:55:39.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: pgmap v142: 97 pgs: 3 remapped+peering, 1 active+recovering+degraded, 4 peering, 2 active+undersized, 5 active+undersized+degraded, 82 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s wr, 527 op/s; 37/1638 objects degraded (2.259%); 0 B/s, 6 objects/s recovering 2024-03-24T11:55:39.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:55:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.309 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:39 smithi032 ceph-mon[66455]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 223739 10.28 MB/sec execute 405 sec latency 1008.522 ms 2024-03-24T11:55:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: pgmap v142: 97 pgs: 3 remapped+peering, 1 active+recovering+degraded, 4 peering, 2 active+undersized, 5 active+undersized+degraded, 82 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s wr, 527 op/s; 37/1638 objects degraded (2.259%); 0 B/s, 6 objects/s recovering 2024-03-24T11:55:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:55:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: failed to encode map e101 with expected crc 2024-03-24T11:55:39.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:39 smithi146 ceph-mon[53008]: failed to encode map e101 with expected crc 2024-03-24T11:55:40.330 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.330 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:55:40.330 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.330 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.331 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.331 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.331 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.331 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:40 smithi032 ceph-mon[66455]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 224048 10.28 MB/sec execute 406 sec latency 819.541 ms 2024-03-24T11:55:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:55:40.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: failed to encode map e102 with expected crc 2024-03-24T11:55:40.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:40 smithi146 ceph-mon[53008]: failed to encode map e102 with expected crc 2024-03-24T11:55:41.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: pgmap v145: 97 pgs: 3 remapped+peering, 1 active+recovering+degraded, 4 peering, 1 active+undersized+remapped, 88 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.3 MiB/s wr, 558 op/s; 5/1623 objects degraded (0.308%); 40 KiB/s, 13 objects/s recovering 2024-03-24T11:55:41.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:55:41.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:41.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:41.467 INFO:teuthology.orchestra.run.smithi032.stdout: 5 224412 10.28 MB/sec execute 407 sec latency 49.328 ms 2024-03-24T11:55:41.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: pgmap v145: 97 pgs: 3 remapped+peering, 1 active+recovering+degraded, 4 peering, 1 active+undersized+remapped, 88 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.3 MiB/s wr, 558 op/s; 5/1623 objects degraded (0.308%); 40 KiB/s, 13 objects/s recovering 2024-03-24T11:55:41.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:55:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: failed to encode map e103 with expected crc 2024-03-24T11:55:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:41.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 224769 10.28 MB/sec execute 408 sec latency 54.226 ms 2024-03-24T11:55:42.578 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.578 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:42.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:42 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 5/1623 objects degraded (0.308%), 1 pg degraded (PG_DEGRADED) 2024-03-24T11:55:42.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:42.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:42.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:55:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:42.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:42 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 5/1623 objects degraded (0.308%), 1 pg degraded (PG_DEGRADED) 2024-03-24T11:55:43.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 225149 10.27 MB/sec execute 409 sec latency 43.350 ms 2024-03-24T11:55:43.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:43 smithi032 ceph-mon[66455]: pgmap v147: 97 pgs: 1 remapped+peering, 1 active+undersized+remapped, 95 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.2 MiB/s wr, 307 op/s; 177 KiB/s, 30 objects/s recovering 2024-03-24T11:55:43.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:43 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:43.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:43 smithi032 ceph-mon[66455]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:55:43.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:43 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/1623 objects degraded (0.308%), 1 pg degraded) 2024-03-24T11:55:43.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:43 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:55:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:43 smithi146 ceph-mon[53008]: pgmap v147: 97 pgs: 1 remapped+peering, 1 active+undersized+remapped, 95 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.2 MiB/s wr, 307 op/s; 177 KiB/s, 30 objects/s recovering 2024-03-24T11:55:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:43 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:43 smithi146 ceph-mon[53008]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:55:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:43 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/1623 objects degraded (0.308%), 1 pg degraded) 2024-03-24T11:55:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:43 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:55:44.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 225376 10.26 MB/sec execute 410 sec latency 91.100 ms 2024-03-24T11:55:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 225607 10.25 MB/sec execute 411 sec latency 69.989 ms 2024-03-24T11:55:45.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:45 smithi032 ceph-mon[66455]: pgmap v148: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.4 MiB/s wr, 270 op/s; 132 KiB/s, 27 objects/s recovering 2024-03-24T11:55:45.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:45 smithi146 ceph-mon[53008]: pgmap v148: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.4 MiB/s wr, 270 op/s; 132 KiB/s, 27 objects/s recovering 2024-03-24T11:55:46.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 225811 10.24 MB/sec execute 412 sec latency 52.702 ms 2024-03-24T11:55:47.478 INFO:teuthology.orchestra.run.smithi032.stdout: 5 226067 10.24 MB/sec execute 413 sec latency 80.965 ms 2024-03-24T11:55:47.777 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:47 smithi032 ceph-mon[66455]: pgmap v149: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.5 MiB/s wr, 206 op/s; 97 KiB/s, 18 objects/s recovering 2024-03-24T11:55:47.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:47 smithi146 ceph-mon[53008]: pgmap v149: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.5 MiB/s wr, 206 op/s; 97 KiB/s, 18 objects/s recovering 2024-03-24T11:55:48.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 226339 10.22 MB/sec execute 414 sec latency 106.404 ms 2024-03-24T11:55:49.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 226766 10.22 MB/sec execute 415 sec latency 37.625 ms 2024-03-24T11:55:49.597 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:49 smithi032 ceph-mon[66455]: pgmap v150: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 457 op/s; 75 KiB/s, 16 objects/s recovering 2024-03-24T11:55:49.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:49 smithi146 ceph-mon[53008]: pgmap v150: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 457 op/s; 75 KiB/s, 16 objects/s recovering 2024-03-24T11:55:50.061 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:55:50.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 227044 10.22 MB/sec execute 416 sec latency 47.580 ms 2024-03-24T11:55:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:55:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 15s ago 15m 22.5M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:55:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (2m) 15s ago 15m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:55:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (2m) 2m ago 14m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:55:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 15s ago 15m 53.6M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:55:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (9m) 15s ago 9m 40.7M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:55:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (9m) 2m ago 9m 16.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (5m) 15s ago 16m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (4m) 2m ago 14m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (4m) 15s ago 16m 48.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (3m) 2m ago 14m 33.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (9m) 15s ago 9m 265M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (9m) 2m ago 9m 88.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (6m) 15s ago 15m 15.7M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (5m) 2m ago 14m 20.1M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (2m) 15s ago 13m 312M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (73s) 15s ago 13m 231M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:55:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (25s) 15s ago 12m 15.1M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:55:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (12m) 15s ago 12m 405M 1937M 17.2.0 e1d6a67b021e b3e360eb7713 2024-03-24T11:55:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (12m) 2m ago 12m 566M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:55:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (11m) 2m ago 11m 397M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:55:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (11m) 2m ago 11m 575M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:55:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (10m) 2m ago 10m 498M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:55:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (4m) 15s ago 14m 66.7M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:55:51.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 227346 10.22 MB/sec execute 417 sec latency 73.117 ms 2024-03-24T11:55:51.626 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:51 smithi032 ceph-mon[66455]: pgmap v151: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 404 op/s; 63 KiB/s, 13 objects/s recovering 2024-03-24T11:55:51.627 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:51 smithi032 ceph-mon[66455]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:51.627 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:51 smithi032 ceph-mon[66455]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:51.627 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:51 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:55:51.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 7 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:55:51.720 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:55:51.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:51 smithi146 ceph-mon[53008]: pgmap v151: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 404 op/s; 63 KiB/s, 13 objects/s recovering 2024-03-24T11:55:51.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:51 smithi146 ceph-mon[53008]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:51.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:51 smithi146 ceph-mon[53008]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:51.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:51 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "9/23 daemons upgraded", 2024-03-24T11:55:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:55:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:55:52.254 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:55:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 227626 10.21 MB/sec execute 418 sec latency 55.541 ms 2024-03-24T11:55:52.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:52 smithi032 ceph-mon[66455]: from='client.34306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:52.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:52 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/4258723608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:52.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:52 smithi146 ceph-mon[53008]: from='client.34306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:52.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:52 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/4258723608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:55:52.906 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:55:53.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 227898 10.21 MB/sec execute 419 sec latency 68.081 ms 2024-03-24T11:55:53.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:53 smithi032 ceph-mon[66455]: pgmap v152: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.5 MiB/s wr, 428 op/s; 55 KiB/s, 11 objects/s recovering 2024-03-24T11:55:53.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:53 smithi032 ceph-mon[66455]: from='client.44235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:53.677 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:53 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/827013296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:55:53.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:53 smithi146 ceph-mon[53008]: pgmap v152: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.5 MiB/s wr, 428 op/s; 55 KiB/s, 11 objects/s recovering 2024-03-24T11:55:53.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:53 smithi146 ceph-mon[53008]: from='client.44235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:55:53.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:53 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/827013296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:55:54.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 228127 10.20 MB/sec execute 420 sec latency 44.774 ms 2024-03-24T11:55:54.696 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:54 smithi032 ceph-mon[66455]: pgmap v153: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 417 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T11:55:54.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:54 smithi146 ceph-mon[53008]: pgmap v153: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 417 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T11:55:55.475 INFO:teuthology.orchestra.run.smithi032.stdout: 5 228400 10.19 MB/sec execute 421 sec latency 55.250 ms 2024-03-24T11:55:56.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 228698 10.18 MB/sec execute 422 sec latency 51.183 ms 2024-03-24T11:55:56.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:56 smithi032 ceph-mon[66455]: pgmap v154: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.9 MiB/s wr, 440 op/s 2024-03-24T11:55:56.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:56.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:56 smithi146 ceph-mon[53008]: pgmap v154: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.9 MiB/s wr, 440 op/s 2024-03-24T11:55:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:55:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:57.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 229027 10.18 MB/sec execute 423 sec latency 47.110 ms 2024-03-24T11:55:57.753 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:57.753 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:57 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:57.753 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:57 smithi032 ceph-mon[66455]: Upgrade: osd.3 is safe to restart 2024-03-24T11:55:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:57 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:55:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:57 smithi146 ceph-mon[53008]: Upgrade: osd.3 is safe to restart 2024-03-24T11:55:58.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 229419 10.18 MB/sec execute 424 sec latency 34.432 ms 2024-03-24T11:55:58.781 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:55:58 smithi032 ceph-mon[66455]: pgmap v155: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.9 MiB/s wr, 488 op/s 2024-03-24T11:55:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:55:58 smithi146 ceph-mon[53008]: pgmap v155: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.9 MiB/s wr, 488 op/s 2024-03-24T11:55:59.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 229709 10.18 MB/sec execute 425 sec latency 52.631 ms 2024-03-24T11:56:00.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 230091 10.17 MB/sec execute 426 sec latency 53.848 ms 2024-03-24T11:56:01.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:01 smithi032 ceph-mon[66455]: pgmap v156: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.7 MiB/s wr, 345 op/s 2024-03-24T11:56:01.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:01 smithi032 ceph-mon[66455]: Upgrade: Updating osd.3 2024-03-24T11:56:01.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:01 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:01.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:01 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:56:01.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:01 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:01.059 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:01 smithi032 ceph-mon[66455]: Deploying daemon osd.3 on smithi032 2024-03-24T11:56:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:01 smithi146 ceph-mon[53008]: pgmap v156: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.7 MiB/s wr, 345 op/s 2024-03-24T11:56:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:01 smithi146 ceph-mon[53008]: Upgrade: Updating osd.3 2024-03-24T11:56:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:01 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:01 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:56:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:01 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:01 smithi146 ceph-mon[53008]: Deploying daemon osd.3 on smithi032 2024-03-24T11:56:01.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 230366 10.16 MB/sec execute 427 sec latency 51.381 ms 2024-03-24T11:56:02.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 230711 10.17 MB/sec execute 428 sec latency 46.415 ms 2024-03-24T11:56:03.100 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:03 smithi032 ceph-mon[66455]: pgmap v157: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 525 op/s 2024-03-24T11:56:03.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:03 smithi146 ceph-mon[53008]: pgmap v157: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 525 op/s 2024-03-24T11:56:03.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 231011 10.17 MB/sec execute 429 sec latency 46.493 ms 2024-03-24T11:56:04.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 231343 10.16 MB/sec execute 430 sec latency 55.688 ms 2024-03-24T11:56:05.143 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:05 smithi032 ceph-mon[66455]: pgmap v158: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.6 MiB/s wr, 470 op/s 2024-03-24T11:56:05.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:05 smithi146 ceph-mon[53008]: pgmap v158: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.6 MiB/s wr, 470 op/s 2024-03-24T11:56:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 231616 10.16 MB/sec execute 431 sec latency 44.670 ms 2024-03-24T11:56:05.916 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:05 smithi032 systemd[1]: Stopping Ceph osd.3 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:56:06.184 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:06 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[46208]: 2024-03-24T11:56:06.095+0000 7f7973027700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:56:06.184 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:06 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[46208]: 2024-03-24T11:56:06.095+0000 7f7973027700 -1 osd.3 103 *** Got signal Terminated *** 2024-03-24T11:56:06.184 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:06 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[46208]: 2024-03-24T11:56:06.095+0000 7f7973027700 -1 osd.3 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:56:06.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 231927 10.15 MB/sec execute 432 sec latency 60.993 ms 2024-03-24T11:56:07.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:07 smithi032 ceph-mon[66455]: pgmap v159: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-03-24T11:56:07.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:07 smithi032 ceph-mon[66455]: osd.3 marked itself down and dead 2024-03-24T11:56:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:07 smithi146 ceph-mon[53008]: pgmap v159: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-03-24T11:56:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:07 smithi146 ceph-mon[53008]: osd.3 marked itself down and dead 2024-03-24T11:56:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 232156 10.14 MB/sec execute 433 sec latency 425.731 ms 2024-03-24T11:56:08.043 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:07 smithi032 podman[80011]: 2024-03-24 11:56:07.709687073 +0000 UTC m=+1.731845456 container died b3e360eb7713021e76bfba52346c75674fb6fc0e3bea404aaddbe936e3a62dd7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, distribution-scope=public, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-24T11:56:08.293 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:56:08.293 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.293 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.294 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.294 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.294 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.294 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.294 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: osdmap e104: 8 total, 7 up, 8 in 2024-03-24T11:56:08.294 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:08 smithi032 ceph-mon[66455]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:56:08.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: osdmap e104: 8 total, 7 up, 8 in 2024-03-24T11:56:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:08 smithi146 ceph-mon[53008]: failed to encode map e104 with expected crc 2024-03-24T11:56:08.476 INFO:teuthology.orchestra.run.smithi032.stdout: 5 232268 10.12 MB/sec execute 434 sec latency 1046.128 ms 2024-03-24T11:56:08.563 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:08 smithi032 podman[80011]: 2024-03-24 11:56:08.334344872 +0000 UTC m=+2.356503249 container cleanup b3e360eb7713021e76bfba52346c75674fb6fc0e3bea404aaddbe936e3a62dd7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, vcs-type=git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , release=754, com.redhat.component=centos-stream-container, architecture=x86_64, name=centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.buildah.version=1.19.8, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T11:56:08.563 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:08 smithi032 bash[80011]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3 2024-03-24T11:56:08.816 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:08 smithi032 podman[80025]: 2024-03-24 11:56:08.601064911 +0000 UTC m=+0.892248388 container remove b3e360eb7713021e76bfba52346c75674fb6fc0e3bea404aaddbe936e3a62dd7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, name=centos-stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_CLEAN=True, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8) 2024-03-24T11:56:09.069 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: pgmap v161: 97 pgs: 16 stale+active+clean, 81 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 553 op/s 2024-03-24T11:56:09.069 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.069 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: osdmap e105: 8 total, 7 up, 8 in 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.070 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:09 smithi032 ceph-mon[66455]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.329 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:09 smithi032 podman[80114]: 2024-03-24 11:56:09.017208495 +0000 UTC m=+0.023556930 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:56:09.329 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:09 smithi032 podman[80114]: 2024-03-24 11:56:09.166416461 +0000 UTC m=+0.172764912 container create 4b8956157ff7cb0c9878ff458ce48b7aab7892b0313420339ebe9a365d754767 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T11:56:09.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: pgmap v161: 97 pgs: 16 stale+active+clean, 81 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 553 op/s 2024-03-24T11:56:09.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: osdmap e105: 8 total, 7 up, 8 in 2024-03-24T11:56:09.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:09 smithi146 ceph-mon[53008]: failed to encode map e105 with expected crc 2024-03-24T11:56:09.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 232573 10.12 MB/sec execute 435 sec latency 54.434 ms 2024-03-24T11:56:09.841 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:09 smithi032 podman[80114]: 2024-03-24 11:56:09.618210828 +0000 UTC m=+0.624559255 container init 4b8956157ff7cb0c9878ff458ce48b7aab7892b0313420339ebe9a365d754767 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T11:56:09.842 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:09 smithi032 podman[80114]: 2024-03-24 11:56:09.62418539 +0000 UTC m=+0.630533819 container start 4b8956157ff7cb0c9878ff458ce48b7aab7892b0313420339ebe9a365d754767 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T11:56:09.842 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:09 smithi032 podman[80114]: 2024-03-24 11:56:09.736474472 +0000 UTC m=+0.742822893 container attach 4b8956157ff7cb0c9878ff458ce48b7aab7892b0313420339ebe9a365d754767 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T11:56:10.092 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:10 smithi032 podman[80114]: 2024-03-24 11:56:10.049193208 +0000 UTC m=+1.055541626 container died 4b8956157ff7cb0c9878ff458ce48b7aab7892b0313420339ebe9a365d754767 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:56:10.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:10 smithi032 ceph-mon[66455]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:56:10.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:10 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 10/1653 objects degraded (0.605%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:10.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:10 smithi146 ceph-mon[53008]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:56:10.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:10 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 10/1653 objects degraded (0.605%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:10.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 232888 10.11 MB/sec execute 436 sec latency 40.241 ms 2024-03-24T11:56:11.355 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:11 smithi032 ceph-mon[66455]: pgmap v163: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 9 peering, 12 stale+active+clean, 72 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.5 MiB/s wr, 338 op/s; 10/1653 objects degraded (0.605%) 2024-03-24T11:56:11.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:11.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:11.356 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:11 smithi032 podman[80171]: 2024-03-24 11:56:11.177782523 +0000 UTC m=+1.122431188 container remove 4b8956157ff7cb0c9878ff458ce48b7aab7892b0313420339ebe9a365d754767 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T11:56:11.356 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:11 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-24T11:56:11.356 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:11 smithi032 systemd[1]: Stopped Ceph osd.3 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:56:11.356 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:11 smithi032 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.3.service: Consumed 4min 37.742s CPU time. 2024-03-24T11:56:11.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:11 smithi146 ceph-mon[53008]: pgmap v163: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 9 peering, 12 stale+active+clean, 72 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.5 MiB/s wr, 338 op/s; 10/1653 objects degraded (0.605%) 2024-03-24T11:56:11.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:11.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 233365 10.12 MB/sec execute 437 sec latency 40.763 ms 2024-03-24T11:56:11.605 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:11 smithi032 systemd[1]: Starting Ceph osd.3 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:56:11.856 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:11 smithi032 podman[80265]: 2024-03-24 11:56:11.748922376 +0000 UTC m=+0.030400530 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:56:12.107 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:11 smithi032 podman[80265]: 2024-03-24 11:56:11.895547457 +0000 UTC m=+0.177025610 container create 7027ab226cac3d0a6e5df0d7a1c1396a40da71b85b024d9382ac70de23a322cc (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:56:12.373 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 podman[80265]: 2024-03-24 11:56:12.313063085 +0000 UTC m=+0.594541246 container init 7027ab226cac3d0a6e5df0d7a1c1396a40da71b85b024d9382ac70de23a322cc (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T11:56:12.373 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 podman[80265]: 2024-03-24 11:56:12.320281918 +0000 UTC m=+0.601760080 container start 7027ab226cac3d0a6e5df0d7a1c1396a40da71b85b024d9382ac70de23a322cc (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:56:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 233800 10.12 MB/sec execute 438 sec latency 30.734 ms 2024-03-24T11:56:12.624 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 podman[80265]: 2024-03-24 11:56:12.50806421 +0000 UTC m=+0.789542370 container attach 7027ab226cac3d0a6e5df0d7a1c1396a40da71b85b024d9382ac70de23a322cc (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T11:56:13.130 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:13 smithi032 ceph-mon[66455]: pgmap v164: 97 pgs: 12 active+undersized, 20 active+undersized+degraded, 9 peering, 56 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 551 op/s; 150/1641 objects degraded (9.141%) 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate[80279]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 bash[80265]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 bash[80265]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 bash[80265]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 bash[80265]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 bash[80265]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 bash[80265]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 bash[80265]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate[80279]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:56:13.131 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate[80279]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:56:13.132 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate[80279]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:56:13.132 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate[80279]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-24T11:56:13.132 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate[80279]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:56:13.132 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate[80279]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-24T11:56:13.132 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:12 smithi032 podman[80265]: 2024-03-24 11:56:12.967848532 +0000 UTC m=+1.249326688 container died 7027ab226cac3d0a6e5df0d7a1c1396a40da71b85b024d9382ac70de23a322cc (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:56:13.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:13 smithi146 ceph-mon[53008]: pgmap v164: 97 pgs: 12 active+undersized, 20 active+undersized+degraded, 9 peering, 56 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 551 op/s; 150/1641 objects degraded (9.141%) 2024-03-24T11:56:13.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 234244 10.13 MB/sec execute 439 sec latency 41.020 ms 2024-03-24T11:56:14.391 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:14 smithi032 ceph-mon[66455]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:56:14.391 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:14 smithi032 podman[80400]: 2024-03-24 11:56:14.187496332 +0000 UTC m=+1.214074597 container remove 7027ab226cac3d0a6e5df0d7a1c1396a40da71b85b024d9382ac70de23a322cc (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T11:56:14.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 234744 10.14 MB/sec execute 440 sec latency 44.120 ms 2024-03-24T11:56:14.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:14 smithi146 ceph-mon[53008]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:56:14.644 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:14 smithi032 podman[80454]: 2024-03-24 11:56:14.382869292 +0000 UTC m=+0.037053646 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:56:14.644 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:14 smithi032 podman[80454]: 2024-03-24 11:56:14.576659383 +0000 UTC m=+0.230843706 container create 05e7bad940288770a4acfec0aa0d381614d3bad272ee2a97ea06d252671f416b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T11:56:15.151 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:14 smithi032 podman[80454]: 2024-03-24 11:56:14.975196144 +0000 UTC m=+0.629380480 container init 05e7bad940288770a4acfec0aa0d381614d3bad272ee2a97ea06d252671f416b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T11:56:15.151 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:14 smithi032 podman[80454]: 2024-03-24 11:56:14.98054084 +0000 UTC m=+0.634725167 container start 05e7bad940288770a4acfec0aa0d381614d3bad272ee2a97ea06d252671f416b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:56:15.151 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:15 smithi032 ceph-osd[80472]: -- 172.21.15.32:0/3091123647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f400fc51e0 con 0x55f400f96c00 2024-03-24T11:56:15.152 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:15 smithi032 bash[80454]: 05e7bad940288770a4acfec0aa0d381614d3bad272ee2a97ea06d252671f416b 2024-03-24T11:56:15.152 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:15 smithi032 systemd[1]: Started Ceph osd.3 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:56:15.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:15 smithi032 ceph-mon[66455]: pgmap v165: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.5 MiB/s wr, 585 op/s; 217/1656 objects degraded (13.104%) 2024-03-24T11:56:15.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 235186 10.15 MB/sec execute 441 sec latency 42.624 ms 2024-03-24T11:56:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:15 smithi146 ceph-mon[53008]: pgmap v165: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.5 MiB/s wr, 585 op/s; 217/1656 objects degraded (13.104%) 2024-03-24T11:56:15.663 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:15 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T11:56:15.567+0000 7ff10c1447c0 -1 Falling back to public interface 2024-03-24T11:56:16.291 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:16.291 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:16.291 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:16.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 235524 10.14 MB/sec execute 442 sec latency 39.467 ms 2024-03-24T11:56:16.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:16.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:16.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:17.299 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:17 smithi032 ceph-mon[66455]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 301 op/s; 217/1656 objects degraded (13.104%) 2024-03-24T11:56:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 235864 10.13 MB/sec execute 443 sec latency 40.061 ms 2024-03-24T11:56:17.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:17 smithi146 ceph-mon[53008]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 301 op/s; 217/1656 objects degraded (13.104%) 2024-03-24T11:56:18.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:18 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 217/1656 objects degraded (13.104%), 27 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:18.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 236155 10.13 MB/sec execute 444 sec latency 58.240 ms 2024-03-24T11:56:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:18 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 217/1656 objects degraded (13.104%), 27 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:19.352 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:19 smithi032 ceph-mon[66455]: pgmap v167: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.7 MiB/s wr, 409 op/s; 210/1686 objects degraded (12.456%) 2024-03-24T11:56:19.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 236573 10.13 MB/sec execute 445 sec latency 30.353 ms 2024-03-24T11:56:19.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:19 smithi146 ceph-mon[53008]: pgmap v167: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.7 MiB/s wr, 409 op/s; 210/1686 objects degraded (12.456%) 2024-03-24T11:56:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 236966 10.13 MB/sec execute 446 sec latency 49.795 ms 2024-03-24T11:56:21.129 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:21 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T11:56:21.043+0000 7ff10c1447c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-24T11:56:21.380 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:21 smithi032 ceph-mon[66455]: pgmap v168: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.5 MiB/s wr, 443 op/s; 201/1635 objects degraded (12.294%); 0 B/s, 0 objects/s recovering 2024-03-24T11:56:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 237356 10.13 MB/sec execute 447 sec latency 40.991 ms 2024-03-24T11:56:21.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:21 smithi146 ceph-mon[53008]: pgmap v168: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.5 MiB/s wr, 443 op/s; 201/1635 objects degraded (12.294%); 0 B/s, 0 objects/s recovering 2024-03-24T11:56:22.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 237679 10.13 MB/sec execute 448 sec latency 43.912 ms 2024-03-24T11:56:22.651 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:22 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T11:56:22.629+0000 7ff10c1447c0 -1 osd.3 103 log_to_monitors true 2024-03-24T11:56:23.414 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:23 smithi032 ceph-mon[66455]: pgmap v169: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 6.3 MiB/s wr, 654 op/s; 201/1620 objects degraded (12.407%); 0 B/s, 0 objects/s recovering 2024-03-24T11:56:23.414 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:23 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 201/1620 objects degraded (12.407%), 27 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:23.414 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:23 smithi032 ceph-mon[66455]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:56:23.435 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:56:23.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 238053 10.14 MB/sec execute 449 sec latency 45.779 ms 2024-03-24T11:56:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:23 smithi146 ceph-mon[53008]: pgmap v169: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 6.3 MiB/s wr, 654 op/s; 201/1620 objects degraded (12.407%); 0 B/s, 0 objects/s recovering 2024-03-24T11:56:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:23 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 201/1620 objects degraded (12.407%), 27 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:23 smithi146 ceph-mon[53008]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:56:23.674 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 11:56:23 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T11:56:23.619+0000 7ff0f9594700 -1 osd.3 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:56:24.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:24 smithi032 ceph-mon[66455]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:56:24.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:24 smithi032 ceph-mon[66455]: osdmap e106: 8 total, 7 up, 8 in 2024-03-24T11:56:24.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:24 smithi032 ceph-mon[66455]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:56:24.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:24 smithi032 ceph-mon[66455]: failed to encode map e106 with expected crc 2024-03-24T11:56:24.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:24 smithi032 ceph-mon[66455]: failed to encode map e106 with expected crc 2024-03-24T11:56:24.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:24 smithi032 ceph-mon[66455]: from='client.44247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:24.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:24 smithi032 ceph-mon[66455]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' 2024-03-24T11:56:24.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 238448 10.14 MB/sec execute 450 sec latency 47.853 ms 2024-03-24T11:56:24.481 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:56:24.481 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 49s ago 15m 22.5M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:56:24.481 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (3m) 49s ago 15m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:56:24.481 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (3m) 3m ago 15m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:56:24.481 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 49s ago 15m 53.6M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:56:24.481 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (10m) 49s ago 10m 40.7M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (10m) 3m ago 10m 16.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (5m) 49s ago 17m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (5m) 3m ago 15m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (4m) 49s ago 17m 48.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (4m) 3m ago 15m 33.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (9m) 49s ago 9m 265M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (9m) 3m ago 9m 88.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (6m) 49s ago 15m 15.7M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (6m) 3m ago 15m 20.1M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (2m) 49s ago 14m 312M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (106s) 49s ago 13m 231M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (58s) 49s ago 13m 15.1M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:56:24.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 starting - - - 1937M 2024-03-24T11:56:24.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (12m) 3m ago 12m 566M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:56:24.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (12m) 3m ago 12m 397M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:56:24.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (11m) 3m ago 11m 575M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:56:24.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (11m) 3m ago 11m 498M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:56:24.483 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 49s ago 14m 66.7M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:56:24.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:24 smithi146 ceph-mon[53008]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:56:24.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:24 smithi146 ceph-mon[53008]: osdmap e106: 8 total, 7 up, 8 in 2024-03-24T11:56:24.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:24 smithi146 ceph-mon[53008]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-03-24T11:56:24.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:24 smithi146 ceph-mon[53008]: failed to encode map e106 with expected crc 2024-03-24T11:56:24.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:24 smithi146 ceph-mon[53008]: failed to encode map e106 with expected crc 2024-03-24T11:56:24.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:24 smithi146 ceph-mon[53008]: from='client.44247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:24.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:24 smithi146 ceph-mon[53008]: from='osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904]' entity='osd.3' 2024-03-24T11:56:24.958 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:56:24.958 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:56:24.958 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:56:24.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:56:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:56:24.960 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "10/23 daemons upgraded", 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:56:25.313 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:56:25.314 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:56:25.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 238495 10.12 MB/sec execute 451 sec latency 893.836 ms 2024-03-24T11:56:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: failed to encode map e106 with expected crc 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: pgmap v171: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.0 MiB/s wr, 588 op/s; 206/1641 objects degraded (12.553%) 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904] boot 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: failed to encode map e107 with expected crc 2024-03-24T11:56:25.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: failed to encode map e107 with expected crc 2024-03-24T11:56:25.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3370309046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:25.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:25.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:25.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: failed to encode map e106 with expected crc 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: pgmap v171: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.0 MiB/s wr, 588 op/s; 206/1641 objects degraded (12.553%) 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: osd.3 [v2:172.21.15.32:6826/1975922904,v1:172.21.15.32:6827/1975922904] boot 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: failed to encode map e107 with expected crc 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: failed to encode map e107 with expected crc 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3370309046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:25.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:25.807 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN Degraded data redundancy: 206/1641 objects degraded (12.553%), 27 pgs degraded 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 206/1641 objects degraded (12.553%), 27 pgs degraded 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.d is active+undersized+degraded, acting [1,4] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.19 is active+undersized+degraded, acting [6,0] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-03-24T11:56:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-03-24T11:56:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-03-24T11:56:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-03-24T11:56:26.477 INFO:teuthology.orchestra.run.smithi032.stdout: 5 238495 10.10 MB/sec execute 452 sec latency 1893.874 ms 2024-03-24T11:56:26.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: failed to encode map e107 with expected crc 2024-03-24T11:56:26.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: failed to encode map e107 with expected crc 2024-03-24T11:56:26.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:26.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:26.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:56:26.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:26.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:26.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:26.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:26 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/4144816655' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:56:26.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: failed to encode map e107 with expected crc 2024-03-24T11:56:26.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: failed to encode map e107 with expected crc 2024-03-24T11:56:26.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:26.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:26.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:56:26.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:26.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:26.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:26.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:26 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/4144816655' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:56:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: 5 238668 10.09 MB/sec execute 453 sec latency 2149.545 ms 2024-03-24T11:56:27.507 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.507 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.507 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: pgmap v174: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 495 op/s; 206/1641 objects degraded (12.553%) 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:56:27.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:27 smithi032 ceph-mon[66455]: failed to encode map e109 with expected crc 2024-03-24T11:56:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e108 with expected crc 2024-03-24T11:56:27.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: pgmap v174: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 495 op/s; 206/1641 objects degraded (12.553%) 2024-03-24T11:56:27.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:27.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:27.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:56:27.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:27 smithi146 ceph-mon[53008]: failed to encode map e109 with expected crc 2024-03-24T11:56:28.269 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:28 smithi032 ceph-mon[66455]: failed to encode map e109 with expected crc 2024-03-24T11:56:28.269 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:28 smithi032 ceph-mon[66455]: failed to encode map e109 with expected crc 2024-03-24T11:56:28.269 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:28 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 206/1641 objects degraded (12.553%), 27 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:28.269 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:28 smithi032 ceph-mon[66455]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:56:28.269 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:28 smithi032 ceph-mon[66455]: failed to encode map e110 with expected crc 2024-03-24T11:56:28.482 INFO:teuthology.orchestra.run.smithi032.stdout: 5 238798 10.07 MB/sec execute 454 sec latency 895.148 ms 2024-03-24T11:56:28.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:28 smithi146 ceph-mon[53008]: failed to encode map e109 with expected crc 2024-03-24T11:56:28.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:28 smithi146 ceph-mon[53008]: failed to encode map e109 with expected crc 2024-03-24T11:56:28.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:28 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 206/1641 objects degraded (12.553%), 27 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:28.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:28 smithi146 ceph-mon[53008]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:56:28.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:28 smithi146 ceph-mon[53008]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: pgmap v177: 97 pgs: 4 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 remapped+peering, 7 active+undersized, 15 active+undersized+degraded, 68 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.1 MiB/s wr, 343 op/s; 207/1605 objects degraded (12.897%); 22 B/s, 4 objects/s recovering 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: failed to encode map e111 with expected crc 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: failed to encode map e111 with expected crc 2024-03-24T11:56:29.317 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:29 smithi032 ceph-mon[66455]: failed to encode map e111 with expected crc 2024-03-24T11:56:29.479 INFO:teuthology.orchestra.run.smithi032.stdout: 5 238991 10.05 MB/sec execute 455 sec latency 946.384 ms 2024-03-24T11:56:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: failed to encode map e110 with expected crc 2024-03-24T11:56:29.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: pgmap v177: 97 pgs: 4 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 remapped+peering, 7 active+undersized, 15 active+undersized+degraded, 68 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.1 MiB/s wr, 343 op/s; 207/1605 objects degraded (12.897%); 22 B/s, 4 objects/s recovering 2024-03-24T11:56:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:56:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: failed to encode map e111 with expected crc 2024-03-24T11:56:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: failed to encode map e111 with expected crc 2024-03-24T11:56:29.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:29 smithi146 ceph-mon[53008]: failed to encode map e111 with expected crc 2024-03-24T11:56:30.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:30 smithi032 ceph-mon[66455]: failed to encode map e111 with expected crc 2024-03-24T11:56:30.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:30 smithi032 ceph-mon[66455]: failed to encode map e111 with expected crc 2024-03-24T11:56:30.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:30 smithi032 ceph-mon[66455]: failed to encode map e111 with expected crc 2024-03-24T11:56:30.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:30 smithi032 ceph-mon[66455]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:56:30.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:30 smithi032 ceph-mon[66455]: failed to encode map e112 with expected crc 2024-03-24T11:56:30.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:30 smithi032 ceph-mon[66455]: failed to encode map e112 with expected crc 2024-03-24T11:56:30.400 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:30 smithi032 ceph-mon[66455]: failed to encode map e112 with expected crc 2024-03-24T11:56:30.479 INFO:teuthology.orchestra.run.smithi032.stdout: 5 239224 10.04 MB/sec execute 456 sec latency 773.936 ms 2024-03-24T11:56:30.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:30 smithi146 ceph-mon[53008]: failed to encode map e111 with expected crc 2024-03-24T11:56:30.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:30 smithi146 ceph-mon[53008]: failed to encode map e111 with expected crc 2024-03-24T11:56:30.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:30 smithi146 ceph-mon[53008]: failed to encode map e111 with expected crc 2024-03-24T11:56:30.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:30 smithi146 ceph-mon[53008]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:56:30.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:30 smithi146 ceph-mon[53008]: failed to encode map e112 with expected crc 2024-03-24T11:56:30.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:30 smithi146 ceph-mon[53008]: failed to encode map e112 with expected crc 2024-03-24T11:56:30.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:30 smithi146 ceph-mon[53008]: failed to encode map e112 with expected crc 2024-03-24T11:56:31.442 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:31 smithi032 ceph-mon[66455]: failed to encode map e112 with expected crc 2024-03-24T11:56:31.442 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:31 smithi032 ceph-mon[66455]: failed to encode map e112 with expected crc 2024-03-24T11:56:31.442 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:31 smithi032 ceph-mon[66455]: pgmap v180: 97 pgs: 5 peering, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 386 op/s; 134/1575 objects degraded (8.508%); 1.2 MiB/s, 21 objects/s recovering 2024-03-24T11:56:31.442 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:31 smithi032 ceph-mon[66455]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:56:31.479 INFO:teuthology.orchestra.run.smithi032.stdout: 5 239405 10.03 MB/sec execute 457 sec latency 821.913 ms 2024-03-24T11:56:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:31 smithi146 ceph-mon[53008]: failed to encode map e112 with expected crc 2024-03-24T11:56:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:31 smithi146 ceph-mon[53008]: failed to encode map e112 with expected crc 2024-03-24T11:56:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:31 smithi146 ceph-mon[53008]: pgmap v180: 97 pgs: 5 peering, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 386 op/s; 134/1575 objects degraded (8.508%); 1.2 MiB/s, 21 objects/s recovering 2024-03-24T11:56:31.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:31 smithi146 ceph-mon[53008]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:56:32.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:32 smithi032 ceph-mon[66455]: failed to encode map e113 with expected crc 2024-03-24T11:56:32.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:32 smithi032 ceph-mon[66455]: failed to encode map e113 with expected crc 2024-03-24T11:56:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: 5 239641 10.03 MB/sec execute 458 sec latency 61.275 ms 2024-03-24T11:56:32.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:32 smithi146 ceph-mon[53008]: failed to encode map e113 with expected crc 2024-03-24T11:56:32.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:32 smithi146 ceph-mon[53008]: failed to encode map e113 with expected crc 2024-03-24T11:56:33.480 INFO:teuthology.orchestra.run.smithi032.stdout: 5 240033 10.03 MB/sec execute 459 sec latency 43.659 ms 2024-03-24T11:56:33.498 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: pgmap v182: 97 pgs: 5 peering, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.4 MiB/s wr, 361 op/s; 73/1554 objects degraded (4.698%); 2.0 MiB/s, 31 objects/s recovering 2024-03-24T11:56:33.498 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.498 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.498 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:33.498 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:33.498 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 73/1554 objects degraded (4.698%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:33.499 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:33 smithi032 ceph-mon[66455]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:56:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: pgmap v182: 97 pgs: 5 peering, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.4 MiB/s wr, 361 op/s; 73/1554 objects degraded (4.698%); 2.0 MiB/s, 31 objects/s recovering 2024-03-24T11:56:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 73/1554 objects degraded (4.698%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:33.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:33 smithi146 ceph-mon[53008]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-24T11:56:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: 5 240391 10.02 MB/sec execute 460 sec latency 46.560 ms 2024-03-24T11:56:34.506 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:34 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/1554 objects degraded (4.698%), 3 pgs degraded) 2024-03-24T11:56:34.506 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:34 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:56:34.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:34 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/1554 objects degraded (4.698%), 3 pgs degraded) 2024-03-24T11:56:34.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:34 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:56:35.480 INFO:teuthology.orchestra.run.smithi032.stdout: 5 240753 10.02 MB/sec execute 461 sec latency 47.088 ms 2024-03-24T11:56:35.513 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:35 smithi032 ceph-mon[66455]: pgmap v183: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 294 op/s; 2.7 MiB/s, 47 objects/s recovering 2024-03-24T11:56:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:35 smithi146 ceph-mon[53008]: pgmap v183: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 294 op/s; 2.7 MiB/s, 47 objects/s recovering 2024-03-24T11:56:36.480 INFO:teuthology.orchestra.run.smithi032.stdout: 5 241095 10.02 MB/sec execute 462 sec latency 48.587 ms 2024-03-24T11:56:37.480 INFO:teuthology.orchestra.run.smithi032.stdout: 5 241629 10.04 MB/sec execute 463 sec latency 33.681 ms 2024-03-24T11:56:37.551 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:37 smithi032 ceph-mon[66455]: pgmap v184: 97 pgs: 97 active+clean; 325 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.2 MiB/s wr, 222 op/s; 2.2 MiB/s, 30 objects/s recovering 2024-03-24T11:56:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:37 smithi146 ceph-mon[53008]: pgmap v184: 97 pgs: 97 active+clean; 325 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.2 MiB/s wr, 222 op/s; 2.2 MiB/s, 30 objects/s recovering 2024-03-24T11:56:38.480 INFO:teuthology.orchestra.run.smithi032.stdout: 5 242060 10.05 MB/sec execute 464 sec latency 32.984 ms 2024-03-24T11:56:39.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 242389 10.05 MB/sec execute 465 sec latency 38.977 ms 2024-03-24T11:56:39.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:39 smithi032 ceph-mon[66455]: pgmap v185: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 457 op/s; 1.4 MiB/s, 25 objects/s recovering 2024-03-24T11:56:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:39 smithi146 ceph-mon[53008]: pgmap v185: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 457 op/s; 1.4 MiB/s, 25 objects/s recovering 2024-03-24T11:56:40.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 242769 10.03 MB/sec execute 466 sec latency 55.909 ms 2024-03-24T11:56:41.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 243074 10.03 MB/sec execute 467 sec latency 50.454 ms 2024-03-24T11:56:41.628 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:41 smithi032 ceph-mon[66455]: pgmap v186: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 424 op/s; 1.2 MiB/s, 21 objects/s recovering 2024-03-24T11:56:41.628 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:41.628 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:41.629 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:41.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:41 smithi146 ceph-mon[53008]: pgmap v186: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 424 op/s; 1.2 MiB/s, 21 objects/s recovering 2024-03-24T11:56:41.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:41.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:41.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 243359 10.02 MB/sec execute 468 sec latency 42.125 ms 2024-03-24T11:56:43.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 243729 10.03 MB/sec execute 469 sec latency 52.556 ms 2024-03-24T11:56:43.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:43 smithi032 ceph-mon[66455]: pgmap v187: 97 pgs: 97 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.0 MiB/s wr, 505 op/s; 1.0 MiB/s, 19 objects/s recovering 2024-03-24T11:56:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:43 smithi146 ceph-mon[53008]: pgmap v187: 97 pgs: 97 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.0 MiB/s wr, 505 op/s; 1.0 MiB/s, 19 objects/s recovering 2024-03-24T11:56:44.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 244098 10.02 MB/sec execute 470 sec latency 73.196 ms 2024-03-24T11:56:45.444 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:45 smithi032 ceph-mon[66455]: pgmap v188: 97 pgs: 97 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.5 MiB/s wr, 391 op/s; 672 KiB/s, 12 objects/s recovering 2024-03-24T11:56:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 244414 10.02 MB/sec execute 471 sec latency 48.008 ms 2024-03-24T11:56:45.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:45 smithi146 ceph-mon[53008]: pgmap v188: 97 pgs: 97 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.5 MiB/s wr, 391 op/s; 672 KiB/s, 12 objects/s recovering 2024-03-24T11:56:46.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 244599 10.01 MB/sec execute 472 sec latency 55.257 ms 2024-03-24T11:56:47.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 244925 10.02 MB/sec execute 473 sec latency 47.990 ms 2024-03-24T11:56:47.522 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:47 smithi032 ceph-mon[66455]: pgmap v189: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 512 op/s 2024-03-24T11:56:47.686 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:47 smithi146 ceph-mon[53008]: pgmap v189: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 512 op/s 2024-03-24T11:56:48.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 245203 10.01 MB/sec execute 474 sec latency 67.672 ms 2024-03-24T11:56:48.541 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:48 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:48.541 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:48 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:48.541 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:48 smithi032 ceph-mon[66455]: Upgrade: osd.4 is safe to restart 2024-03-24T11:56:48.607 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:48 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:48.607 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:48 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:56:48.607 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:48 smithi146 ceph-mon[53008]: Upgrade: osd.4 is safe to restart 2024-03-24T11:56:49.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 245490 10.01 MB/sec execute 475 sec latency 65.880 ms 2024-03-24T11:56:49.566 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:49 smithi032 ceph-mon[66455]: pgmap v190: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.7 MiB/s wr, 593 op/s 2024-03-24T11:56:49.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:49 smithi146 ceph-mon[53008]: pgmap v190: 97 pgs: 97 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.7 MiB/s wr, 593 op/s 2024-03-24T11:56:50.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 245687 10.00 MB/sec execute 476 sec latency 54.471 ms 2024-03-24T11:56:51.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:50 smithi032 ceph-mon[66455]: pgmap v191: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 464 op/s 2024-03-24T11:56:51.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:50 smithi032 ceph-mon[66455]: Upgrade: Updating osd.4 2024-03-24T11:56:51.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:51.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:56:51.106 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:51.107 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:50 smithi032 ceph-mon[66455]: Deploying daemon osd.4 on smithi146 2024-03-24T11:56:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:50 smithi146 ceph-mon[53008]: pgmap v191: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 464 op/s 2024-03-24T11:56:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:50 smithi146 ceph-mon[53008]: Upgrade: Updating osd.4 2024-03-24T11:56:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:56:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:51.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:50 smithi146 ceph-mon[53008]: Deploying daemon osd.4 on smithi146 2024-03-24T11:56:51.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 245964 9.99 MB/sec execute 477 sec latency 55.988 ms 2024-03-24T11:56:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 246256 9.98 MB/sec execute 478 sec latency 66.581 ms 2024-03-24T11:56:53.185 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:52 smithi032 ceph-mon[66455]: pgmap v192: 97 pgs: 97 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.6 MiB/s wr, 569 op/s 2024-03-24T11:56:53.357 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:52 smithi146 ceph-mon[53008]: pgmap v192: 97 pgs: 97 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.6 MiB/s wr, 569 op/s 2024-03-24T11:56:53.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 246526 9.97 MB/sec execute 479 sec latency 64.644 ms 2024-03-24T11:56:54.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:53 smithi146 systemd[1]: Stopping Ceph osd.4 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:56:54.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 246736 9.97 MB/sec execute 480 sec latency 59.075 ms 2024-03-24T11:56:54.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[31212]: 2024-03-24T11:56:54.163+0000 7fc2d97cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:56:54.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[31212]: 2024-03-24T11:56:54.163+0000 7fc2d97cc700 -1 osd.4 113 *** Got signal Terminated *** 2024-03-24T11:56:54.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[31212]: 2024-03-24T11:56:54.163+0000 7fc2d97cc700 -1 osd.4 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:56:55.083 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:54 smithi032 ceph-mon[66455]: pgmap v193: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-03-24T11:56:55.083 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:54 smithi032 ceph-mon[66455]: osd.4 marked itself down and dead 2024-03-24T11:56:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:54 smithi146 ceph-mon[53008]: pgmap v193: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-03-24T11:56:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:54 smithi146 ceph-mon[53008]: osd.4 marked itself down and dead 2024-03-24T11:56:55.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 246920 9.95 MB/sec execute 481 sec latency 485.387 ms 2024-03-24T11:56:55.884 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:55 smithi146 podman[57490]: 2024-03-24 11:56:55.617806163 +0000 UTC m=+1.593846067 container died eebb612477a0dc79c07f728be4a3ba62e22b15237c5b3b97c0fb0a34970b99cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_CLEAN=True, io.buildah.version=1.19.8, release=754, version=8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-03-24T11:56:56.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:56:56.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: failed to encode map e114 with expected crc 2024-03-24T11:56:56.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: osdmap e114: 8 total, 7 up, 8 in 2024-03-24T11:56:56.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 4]}]: dispatch 2024-03-24T11:56:56.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]: dispatch 2024-03-24T11:56:56.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5, 3, 4]}]: dispatch 2024-03-24T11:56:56.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 4, 1, 2]}]: dispatch 2024-03-24T11:56:56.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 4]}]: dispatch 2024-03-24T11:56:56.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 4]}]: dispatch 2024-03-24T11:56:56.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 4]}]: dispatch 2024-03-24T11:56:56.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:56.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:56.270 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: failed to encode map e114 with expected crc 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: osdmap e114: 8 total, 7 up, 8 in 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 4]}]: dispatch 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]: dispatch 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5, 3, 4]}]: dispatch 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 4, 1, 2]}]: dispatch 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 4]}]: dispatch 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 4]}]: dispatch 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 4]}]: dispatch 2024-03-24T11:56:56.274 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:56:56.275 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:56.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 247157 9.95 MB/sec execute 482 sec latency 1046.608 ms 2024-03-24T11:56:56.790 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:56 smithi146 podman[57490]: 2024-03-24 11:56:56.530648418 +0000 UTC m=+2.506688339 container cleanup eebb612477a0dc79c07f728be4a3ba62e22b15237c5b3b97c0fb0a34970b99cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, version=8, RELEASE=HEAD, architecture=x86_64, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, distribution-scope=public) 2024-03-24T11:56:56.790 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:56 smithi146 bash[57490]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4 2024-03-24T11:56:57.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: pgmap v195: 97 pgs: 14 stale+active+clean, 83 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 358 op/s 2024-03-24T11:56:57.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 4]}]': finished 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]': finished 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5, 3, 4]}]': finished 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 4, 1, 2]}]': finished 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 4]}]': finished 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 4]}]': finished 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 4]}]': finished 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: osdmap e115: 8 total, 7 up, 8 in 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:56 smithi032 ceph-mon[66455]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:57.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:56 smithi146 podman[57507]: 2024-03-24 11:56:56.790659746 +0000 UTC m=+1.169962813 container remove eebb612477a0dc79c07f728be4a3ba62e22b15237c5b3b97c0fb0a34970b99cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, ceph=True, RELEASE=HEAD, name=centos-stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, release=754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: pgmap v195: 97 pgs: 14 stale+active+clean, 83 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 358 op/s 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 4]}]': finished 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]': finished 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5, 3, 4]}]': finished 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 4, 1, 2]}]': finished 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 4]}]': finished 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 4]}]': finished 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 4]}]': finished 2024-03-24T11:56:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: osdmap e115: 8 total, 7 up, 8 in 2024-03-24T11:56:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: failed to encode map e115 with expected crc 2024-03-24T11:56:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:56 smithi146 ceph-mon[53008]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:57.184 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:56:57.184 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 32s ago 16m 22.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:56:57.184 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (3m) 32s ago 16m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:56:57.184 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (3m) 3m ago 15m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:56:57.184 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 32s ago 16m 52.0M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (10m) 32s ago 10m 40.8M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (10m) 3m ago 10m 16.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (6m) 32s ago 17m 507M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (5m) 3m ago 15m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (5m) 32s ago 17m 50.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (4m) 3m ago 15m 33.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (10m) 32s ago 10m 279M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (10m) 3m ago 10m 88.5M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 32s ago 16m 16.1M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (6m) 3m ago 15m 20.1M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 32s ago 15m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (2m) 32s ago 14m 285M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (91s) 32s ago 14m 217M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:56:57.185 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (42s) 32s ago 13m 14.9M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T11:56:57.186 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (13m) 3m ago 13m 566M 2704M 17.2.0 e1d6a67b021e eebb612477a0 2024-03-24T11:56:57.186 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (12m) 3m ago 12m 397M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:56:57.186 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (12m) 3m ago 12m 575M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:56:57.186 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (12m) 3m ago 12m 498M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:56:57.186 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 32s ago 15m 68.8M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:56:57.486 INFO:teuthology.orchestra.run.smithi032.stdout: 5 247471 9.94 MB/sec execute 483 sec latency 63.975 ms 2024-03-24T11:56:57.494 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:57 smithi146 podman[57590]: 2024-03-24 11:56:57.087340679 +0000 UTC m=+0.022281743 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:56:57.494 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:57 smithi146 podman[57590]: 2024-03-24 11:56:57.225214919 +0000 UTC m=+0.160155972 container create d866419b92aa20af639d2b4e921c9b5dd0b282f55a93d9bd70458973ae8942b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T11:56:57.494 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:57 smithi146 podman[57590]: 2024-03-24 11:56:57.494687808 +0000 UTC m=+0.429628880 container init d866419b92aa20af639d2b4e921c9b5dd0b282f55a93d9bd70458973ae8942b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:56:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:56:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:56:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:56:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:56:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:56:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:56:57.810 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:56:57.817 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:57 smithi146 podman[57590]: 2024-03-24 11:56:57.498336751 +0000 UTC m=+0.433277812 container start d866419b92aa20af639d2b4e921c9b5dd0b282f55a93d9bd70458973ae8942b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T11:56:57.817 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:57 smithi146 podman[57590]: 2024-03-24 11:56:57.594931125 +0000 UTC m=+0.529872187 container attach d866419b92aa20af639d2b4e921c9b5dd0b282f55a93d9bd70458973ae8942b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T11:56:58.016 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:58 smithi032 ceph-mon[66455]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:58.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:58 smithi032 ceph-mon[66455]: failed to encode map e116 with expected crc 2024-03-24T11:56:58.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:58 smithi032 ceph-mon[66455]: osdmap e116: 8 total, 7 up, 8 in 2024-03-24T11:56:58.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:58 smithi032 ceph-mon[66455]: failed to encode map e116 with expected crc 2024-03-24T11:56:58.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:58 smithi032 ceph-mon[66455]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:58.058 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:58 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3434658943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:58.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:57 smithi146 podman[57590]: 2024-03-24 11:56:57.818502837 +0000 UTC m=+0.753443888 container died d866419b92aa20af639d2b4e921c9b5dd0b282f55a93d9bd70458973ae8942b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T11:56:58.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:58 smithi146 ceph-mon[53008]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:58 smithi146 ceph-mon[53008]: failed to encode map e116 with expected crc 2024-03-24T11:56:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:58 smithi146 ceph-mon[53008]: osdmap e116: 8 total, 7 up, 8 in 2024-03-24T11:56:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:58 smithi146 ceph-mon[53008]: failed to encode map e116 with expected crc 2024-03-24T11:56:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:58 smithi146 ceph-mon[53008]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:58.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:58 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3434658943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:56:58.379 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:56:58.379 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:56:58.379 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:56:58.379 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "10/23 daemons upgraded", 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:56:58.380 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:56:58.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 247726 9.93 MB/sec execute 484 sec latency 63.003 ms 2024-03-24T11:56:58.884 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:58 smithi146 podman[57616]: 2024-03-24 11:56:58.501549667 +0000 UTC m=+0.677448909 container remove d866419b92aa20af639d2b4e921c9b5dd0b282f55a93d9bd70458973ae8942b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T11:56:58.884 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:58 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-24T11:56:58.884 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:58 smithi146 systemd[1]: Stopped Ceph osd.4 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:56:58.885 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:58 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.4.service: Consumed 4min 47.487s CPU time. 2024-03-24T11:56:58.885 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:58 smithi146 systemd[1]: Starting Ceph osd.4 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:56:59.034 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 134/1578 objects degraded (8.492%), 16 pgs degraded 2024-03-24T11:56:59.034 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T11:56:59.034 INFO:teuthology.orchestra.run.smithi032.stdout: osd.4 (root=default,host=smithi146) is down 2024-03-24T11:56:59.034 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 134/1578 objects degraded (8.492%), 16 pgs degraded 2024-03-24T11:56:59.034 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-03-24T11:56:59.034 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-03-24T11:56:59.034 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.b is active+undersized+degraded, acting [7,5] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-03-24T11:56:59.035 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-03-24T11:56:59.061 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:59 smithi032 ceph-mon[66455]: pgmap v198: 97 pgs: 4 active+undersized, 16 active+undersized+degraded, 8 peering, 7 stale+active+clean, 62 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.6 MiB/s wr, 305 op/s; 134/1578 objects degraded (8.492%) 2024-03-24T11:56:59.061 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:59 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 134/1578 objects degraded (8.492%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:59.061 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:59 smithi032 ceph-mon[66455]: osdmap e117: 8 total, 7 up, 8 in 2024-03-24T11:56:59.062 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:59 smithi032 ceph-mon[66455]: failed to encode map e117 with expected crc 2024-03-24T11:56:59.062 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:59 smithi032 ceph-mon[66455]: failed to encode map e117 with expected crc 2024-03-24T11:56:59.062 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:56:59 smithi032 ceph-mon[66455]: from='client.44283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:59.247 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-mon[53008]: pgmap v198: 97 pgs: 4 active+undersized, 16 active+undersized+degraded, 8 peering, 7 stale+active+clean, 62 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.6 MiB/s wr, 305 op/s; 134/1578 objects degraded (8.492%) 2024-03-24T11:56:59.247 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 134/1578 objects degraded (8.492%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T11:56:59.247 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-mon[53008]: osdmap e117: 8 total, 7 up, 8 in 2024-03-24T11:56:59.247 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-mon[53008]: failed to encode map e117 with expected crc 2024-03-24T11:56:59.248 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-mon[53008]: failed to encode map e117 with expected crc 2024-03-24T11:56:59.248 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-mon[53008]: from='client.44283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:56:59.248 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:58 smithi146 podman[57717]: 2024-03-24 11:56:58.854655088 +0000 UTC m=+0.018057040 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:56:59.248 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:58 smithi146 podman[57717]: 2024-03-24 11:56:58.977690104 +0000 UTC m=+0.141092077 container create 1b2fe14da2f757c00e6634aff82f98bd6d0a870e73dba9e1a1ce8e1cce657285 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-24T11:56:59.248 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 podman[57717]: 2024-03-24 11:56:59.247832657 +0000 UTC m=+0.411234599 container init 1b2fe14da2f757c00e6634aff82f98bd6d0a870e73dba9e1a1ce8e1cce657285 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T11:56:59.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 248045 9.93 MB/sec execute 485 sec latency 54.288 ms 2024-03-24T11:56:59.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 podman[57717]: 2024-03-24 11:56:59.251877543 +0000 UTC m=+0.415279482 container start 1b2fe14da2f757c00e6634aff82f98bd6d0a870e73dba9e1a1ce8e1cce657285 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True) 2024-03-24T11:56:59.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 podman[57717]: 2024-03-24 11:56:59.347294584 +0000 UTC m=+0.510696516 container attach 1b2fe14da2f757c00e6634aff82f98bd6d0a870e73dba9e1a1ce8e1cce657285 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate[57731]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate[57731]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 bash[57717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 bash[57717]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 bash[57717]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 bash[57717]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 bash[57717]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 bash[57717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 bash[57717]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-24T11:57:00.016 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate[57731]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:57:00.017 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate[57731]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:57:00.017 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate[57731]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-24T11:57:00.017 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate[57731]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:57:00.017 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate[57731]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-24T11:57:00.017 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:56:59 smithi146 podman[57717]: 2024-03-24 11:56:59.738397746 +0000 UTC m=+0.901799695 container died 1b2fe14da2f757c00e6634aff82f98bd6d0a870e73dba9e1a1ce8e1cce657285 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T11:57:00.085 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:00 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3554836184' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:57:00.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:00 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3554836184' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:57:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: 5 248274 9.92 MB/sec execute 486 sec latency 62.132 ms 2024-03-24T11:57:00.782 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:00 smithi146 podman[57863]: 2024-03-24 11:57:00.418750113 +0000 UTC m=+0.675127343 container remove 1b2fe14da2f757c00e6634aff82f98bd6d0a870e73dba9e1a1ce8e1cce657285 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T11:57:00.782 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:00 smithi146 podman[57929]: 2024-03-24 11:57:00.533569392 +0000 UTC m=+0.018270365 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:57:00.782 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:00 smithi146 podman[57929]: 2024-03-24 11:57:00.653027654 +0000 UTC m=+0.137728621 container create 84c8e53a61a864543b833873eef52e76346f8dce868959299c61959c03e6e13b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T11:57:01.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:01 smithi032 ceph-mon[66455]: pgmap v200: 97 pgs: 4 active+clean+remapped, 1 remapped+peering, 10 active+undersized, 25 active+undersized+degraded, 7 peering, 50 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 473 op/s; 213/1524 objects degraded (13.976%); 1/1524 objects misplaced (0.066%); 3 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:01.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:01 smithi146 ceph-mon[53008]: pgmap v200: 97 pgs: 4 active+clean+remapped, 1 remapped+peering, 10 active+undersized, 25 active+undersized+degraded, 7 peering, 50 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 473 op/s; 213/1524 objects degraded (13.976%); 1/1524 objects misplaced (0.066%); 3 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:01.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:00 smithi146 podman[57929]: 2024-03-24 11:57:00.914356315 +0000 UTC m=+0.399057293 container init 84c8e53a61a864543b833873eef52e76346f8dce868959299c61959c03e6e13b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:57:01.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:00 smithi146 podman[57929]: 2024-03-24 11:57:00.91768579 +0000 UTC m=+0.402386765 container start 84c8e53a61a864543b833873eef52e76346f8dce868959299c61959c03e6e13b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T11:57:01.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:00 smithi146 ceph-osd[57948]: -- 172.21.15.146:0/2067727435 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d4bbc591e0 con 0x55d4bbc2ac00 2024-03-24T11:57:01.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:01 smithi146 bash[57929]: 84c8e53a61a864543b833873eef52e76346f8dce868959299c61959c03e6e13b 2024-03-24T11:57:01.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:01 smithi146 systemd[1]: Started Ceph osd.4 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:57:01.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 248604 9.93 MB/sec execute 487 sec latency 69.889 ms 2024-03-24T11:57:01.884 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:01 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T11:57:01.468+0000 7fbb1a1567c0 -1 Falling back to public interface 2024-03-24T11:57:02.147 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:02.147 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:02.147 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:02.241 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:02 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:02.242 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:02 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:02.242 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:02 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:02.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 248858 9.92 MB/sec execute 488 sec latency 68.805 ms 2024-03-24T11:57:03.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 249074 9.91 MB/sec execute 489 sec latency 47.569 ms 2024-03-24T11:57:03.699 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:03 smithi032 ceph-mon[66455]: pgmap v201: 97 pgs: 6 active+clean+remapped, 1 remapped+peering, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.3 MiB/s wr, 480 op/s; 257/1569 objects degraded (16.380%); 1/1569 objects misplaced (0.064%); 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:03.837 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:03 smithi146 ceph-mon[53008]: pgmap v201: 97 pgs: 6 active+clean+remapped, 1 remapped+peering, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.3 MiB/s wr, 480 op/s; 257/1569 objects degraded (16.380%); 1/1569 objects misplaced (0.064%); 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:04.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 249294 9.92 MB/sec execute 490 sec latency 43.518 ms 2024-03-24T11:57:04.741 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:04 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 270/1632 objects degraded (16.544%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:04.838 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:04 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 270/1632 objects degraded (16.544%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:05.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 249523 9.90 MB/sec execute 491 sec latency 70.795 ms 2024-03-24T11:57:05.748 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:05 smithi032 ceph-mon[66455]: pgmap v202: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 402 op/s; 270/1632 objects degraded (16.544%); 1/1632 objects misplaced (0.061%); 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:05.748 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:05 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:05.748 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:05 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:05.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:05 smithi146 ceph-mon[53008]: pgmap v202: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 402 op/s; 270/1632 objects degraded (16.544%); 1/1632 objects misplaced (0.061%); 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:05.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:05 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:05.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:05 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:06.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 249860 9.90 MB/sec execute 492 sec latency 89.168 ms 2024-03-24T11:57:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 250274 9.90 MB/sec execute 493 sec latency 26.757 ms 2024-03-24T11:57:07.593 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:07 smithi146 ceph-mon[53008]: pgmap v203: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.3 KiB/s rd, 3.5 MiB/s wr, 329 op/s; 273/1644 objects degraded (16.606%); 1/1644 objects misplaced (0.061%); 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:07.593 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:07.593 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:07.771 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:07 smithi032 ceph-mon[66455]: pgmap v203: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.3 KiB/s rd, 3.5 MiB/s wr, 329 op/s; 273/1644 objects degraded (16.606%); 1/1644 objects misplaced (0.061%); 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:07.771 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:07.772 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:08.353 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:07 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T11:57:07.902+0000 7fbb1a1567c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-24T11:57:08.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 250709 9.90 MB/sec execute 494 sec latency 30.522 ms 2024-03-24T11:57:09.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 251036 9.89 MB/sec execute 495 sec latency 60.390 ms 2024-03-24T11:57:09.552 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:09 smithi032 ceph-mon[66455]: pgmap v204: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.0 MiB/s wr, 347 op/s; 293/1707 objects degraded (17.165%); 1/1707 objects misplaced (0.059%); 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:09.552 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:09 smithi032 ceph-mon[66455]: from='osd.4 [v2:172.21.15.146:6800/860292253,v1:172.21.15.146:6801/860292253]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:57:09.552 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:09 smithi032 ceph-mon[66455]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:57:09.741 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:09 smithi146 ceph-mon[53008]: pgmap v204: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.0 MiB/s wr, 347 op/s; 293/1707 objects degraded (17.165%); 1/1707 objects misplaced (0.059%); 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T11:57:09.741 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:09 smithi146 ceph-mon[53008]: from='osd.4 [v2:172.21.15.146:6800/860292253,v1:172.21.15.146:6801/860292253]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:57:09.742 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:09 smithi146 ceph-mon[53008]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:57:09.742 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T11:57:09.425+0000 7fbb1a1567c0 -1 osd.4 113 log_to_monitors true 2024-03-24T11:57:10.384 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T11:57:10.025+0000 7fbb075a6700 -1 osd.4 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:57:10.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 251274 9.88 MB/sec execute 496 sec latency 245.862 ms 2024-03-24T11:57:10.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:57:10.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: from='osd.4 [v2:172.21.15.146:6800/860292253,v1:172.21.15.146:6801/860292253]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:10.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: osdmap e118: 8 total, 7 up, 8 in 2024-03-24T11:57:10.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:10.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: failed to encode map e118 with expected crc 2024-03-24T11:57:10.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: failed to encode map e118 with expected crc 2024-03-24T11:57:10.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: failed to encode map e118 with expected crc 2024-03-24T11:57:10.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:10 smithi032 ceph-mon[66455]: from='osd.4 ' entity='osd.4' 2024-03-24T11:57:10.842 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:57:10.842 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: from='osd.4 [v2:172.21.15.146:6800/860292253,v1:172.21.15.146:6801/860292253]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:10.842 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: osdmap e118: 8 total, 7 up, 8 in 2024-03-24T11:57:10.843 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:10.843 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: failed to encode map e118 with expected crc 2024-03-24T11:57:10.843 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: failed to encode map e118 with expected crc 2024-03-24T11:57:10.843 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: failed to encode map e118 with expected crc 2024-03-24T11:57:10.843 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:10 smithi146 ceph-mon[53008]: from='osd.4 ' entity='osd.4' 2024-03-24T11:57:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 251526 9.88 MB/sec execute 497 sec latency 459.725 ms 2024-03-24T11:57:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: pgmap v206: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.2 MiB/s wr, 369 op/s; 290/1713 objects degraded (16.929%); 1/1713 objects misplaced (0.058%) 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 290/1713 objects degraded (16.929%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: failed to encode map e119 with expected crc 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: osd.4 [v2:172.21.15.146:6800/860292253,v1:172.21.15.146:6801/860292253] boot 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:11.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: pgmap v206: 97 pgs: 7 active+clean+remapped, 12 active+undersized, 29 active+undersized+degraded, 49 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.2 MiB/s wr, 369 op/s; 290/1713 objects degraded (16.929%); 1/1713 objects misplaced (0.058%) 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 290/1713 objects degraded (16.929%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:11.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: failed to encode map e119 with expected crc 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: osd.4 [v2:172.21.15.146:6800/860292253,v1:172.21.15.146:6801/860292253] boot 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:11.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:11.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 251639 9.87 MB/sec execute 498 sec latency 1089.020 ms 2024-03-24T11:57:12.640 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:12.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T11:57:12.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:57:12.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:12 smithi032 ceph-mon[66455]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:12.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T11:57:12.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:57:12.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: failed to encode map e120 with expected crc 2024-03-24T11:57:12.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:12 smithi146 ceph-mon[53008]: failed to encode map e120 with expected crc 2024-03-24T11:57:13.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 251975 9.86 MB/sec execute 499 sec latency 880.992 ms 2024-03-24T11:57:13.657 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: pgmap v208: 97 pgs: 2 remapped+peering, 15 peering, 5 active+clean+remapped, 12 active+undersized, 14 active+undersized+degraded, 49 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 582 op/s; 137/1632 objects degraded (8.395%); 1/1632 objects misplaced (0.061%) 2024-03-24T11:57:13.657 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.657 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:57:13.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:13 smithi032 ceph-mon[66455]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: pgmap v208: 97 pgs: 2 remapped+peering, 15 peering, 5 active+clean+remapped, 12 active+undersized, 14 active+undersized+degraded, 49 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 582 op/s; 137/1632 objects degraded (8.395%); 1/1632 objects misplaced (0.061%) 2024-03-24T11:57:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:57:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: failed to encode map e121 with expected crc 2024-03-24T11:57:13.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:13 smithi146 ceph-mon[53008]: failed to encode map e121 with expected crc 2024-03-24T11:57:14.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 252256 9.86 MB/sec execute 500 sec latency 62.386 ms 2024-03-24T11:57:15.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:15 smithi032 ceph-mon[66455]: pgmap v211: 97 pgs: 3 active+recovery_wait+degraded, 2 remapped+peering, 16 peering, 4 active+clean+remapped, 10 active+undersized, 8 active+undersized+degraded, 54 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.4 MiB/s wr, 532 op/s; 114/1581 objects degraded (7.211%); 1/1581 objects misplaced (0.063%); 133 KiB/s, 4 objects/s recovering 2024-03-24T11:57:15.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:15 smithi032 ceph-mon[66455]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:15 smithi032 ceph-mon[66455]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.185 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:15 smithi032 ceph-mon[66455]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:57:15.185 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:15 smithi032 ceph-mon[66455]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.185 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:15 smithi032 ceph-mon[66455]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:15 smithi146 ceph-mon[53008]: pgmap v211: 97 pgs: 3 active+recovery_wait+degraded, 2 remapped+peering, 16 peering, 4 active+clean+remapped, 10 active+undersized, 8 active+undersized+degraded, 54 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.4 MiB/s wr, 532 op/s; 114/1581 objects degraded (7.211%); 1/1581 objects misplaced (0.063%); 133 KiB/s, 4 objects/s recovering 2024-03-24T11:57:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:15 smithi146 ceph-mon[53008]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:15 smithi146 ceph-mon[53008]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:15 smithi146 ceph-mon[53008]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:57:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:15 smithi146 ceph-mon[53008]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:15 smithi146 ceph-mon[53008]: failed to encode map e122 with expected crc 2024-03-24T11:57:15.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 252548 9.87 MB/sec execute 501 sec latency 618.668 ms 2024-03-24T11:57:16.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:16 smithi032 ceph-mon[66455]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:16 smithi032 ceph-mon[66455]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:16 smithi032 ceph-mon[66455]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:57:16.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:16 smithi032 ceph-mon[66455]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:16 smithi032 ceph-mon[66455]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:16 smithi032 ceph-mon[66455]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:16 smithi032 ceph-mon[66455]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:16 smithi146 ceph-mon[53008]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:16 smithi146 ceph-mon[53008]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:16 smithi146 ceph-mon[53008]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:57:16.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:16 smithi146 ceph-mon[53008]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:16 smithi146 ceph-mon[53008]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:16 smithi146 ceph-mon[53008]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:16 smithi146 ceph-mon[53008]: failed to encode map e123 with expected crc 2024-03-24T11:57:16.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 252808 9.86 MB/sec execute 502 sec latency 927.878 ms 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: pgmap v214: 97 pgs: 3 active+recovery_wait+degraded, 1 remapped+peering, 17 peering, 2 active+undersized, 3 active+undersized+degraded, 71 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.4 MiB/s wr, 88 op/s; 67/1581 objects degraded (4.238%); 398 KiB/s, 21 objects/s recovering 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 67/1581 objects degraded (4.238%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:57:17.211 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.212 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.212 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.212 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:17 smithi032 ceph-mon[66455]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: pgmap v214: 97 pgs: 3 active+recovery_wait+degraded, 1 remapped+peering, 17 peering, 2 active+undersized, 3 active+undersized+degraded, 71 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.4 MiB/s wr, 88 op/s; 67/1581 objects degraded (4.238%); 398 KiB/s, 21 objects/s recovering 2024-03-24T11:57:17.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 67/1581 objects degraded (4.238%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:17.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:57:17.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:17 smithi146 ceph-mon[53008]: failed to encode map e124 with expected crc 2024-03-24T11:57:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 253031 9.85 MB/sec execute 503 sec latency 862.005 ms 2024-03-24T11:57:18.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:18 smithi032 ceph-mon[66455]: failed to encode map e125 with expected crc 2024-03-24T11:57:18.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:18 smithi032 ceph-mon[66455]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:57:18.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:18 smithi032 ceph-mon[66455]: failed to encode map e125 with expected crc 2024-03-24T11:57:18.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:18 smithi032 ceph-mon[66455]: failed to encode map e125 with expected crc 2024-03-24T11:57:18.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:18 smithi146 ceph-mon[53008]: failed to encode map e125 with expected crc 2024-03-24T11:57:18.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:18 smithi146 ceph-mon[53008]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:57:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:18 smithi146 ceph-mon[53008]: failed to encode map e125 with expected crc 2024-03-24T11:57:18.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:18 smithi146 ceph-mon[53008]: failed to encode map e125 with expected crc 2024-03-24T11:57:18.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 253325 9.85 MB/sec execute 504 sec latency 63.690 ms 2024-03-24T11:57:19.269 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:19 smithi032 ceph-mon[66455]: pgmap v217: 97 pgs: 3 active+recovery_wait+degraded, 3 peering, 91 active+clean; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.8 MiB/s wr, 246 op/s; 45/1569 objects degraded (2.868%); 1.3 MiB/s, 67 keys/s, 57 objects/s recovering 2024-03-24T11:57:19.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:19 smithi146 ceph-mon[53008]: pgmap v217: 97 pgs: 3 active+recovery_wait+degraded, 3 peering, 91 active+clean; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.8 MiB/s wr, 246 op/s; 45/1569 objects degraded (2.868%); 1.3 MiB/s, 67 keys/s, 57 objects/s recovering 2024-03-24T11:57:19.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 253575 9.84 MB/sec execute 505 sec latency 70.415 ms 2024-03-24T11:57:20.286 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:20 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/1569 objects degraded (2.868%), 3 pgs degraded) 2024-03-24T11:57:20.286 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:20 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:57:20.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:20 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/1569 objects degraded (2.868%), 3 pgs degraded) 2024-03-24T11:57:20.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:20 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:57:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 253870 9.83 MB/sec execute 506 sec latency 69.347 ms 2024-03-24T11:57:21.299 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:21 smithi032 ceph-mon[66455]: pgmap v218: 97 pgs: 1 peering, 96 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.6 MiB/s wr, 413 op/s; 983 KiB/s, 47 keys/s, 48 objects/s recovering 2024-03-24T11:57:21.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:21 smithi146 ceph-mon[53008]: pgmap v218: 97 pgs: 1 peering, 96 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.6 MiB/s wr, 413 op/s; 983 KiB/s, 47 keys/s, 48 objects/s recovering 2024-03-24T11:57:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 254201 9.83 MB/sec execute 507 sec latency 61.644 ms 2024-03-24T11:57:22.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 254690 9.83 MB/sec execute 508 sec latency 39.343 ms 2024-03-24T11:57:23.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:23 smithi032 ceph-mon[66455]: pgmap v219: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.4 MiB/s wr, 556 op/s; 679 KiB/s, 40 keys/s, 30 objects/s recovering 2024-03-24T11:57:23.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:23 smithi146 ceph-mon[53008]: pgmap v219: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.4 MiB/s wr, 556 op/s; 679 KiB/s, 40 keys/s, 30 objects/s recovering 2024-03-24T11:57:23.494 INFO:teuthology.orchestra.run.smithi032.stdout: 5 255151 9.85 MB/sec execute 509 sec latency 37.646 ms 2024-03-24T11:57:24.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 255720 9.86 MB/sec execute 510 sec latency 39.226 ms 2024-03-24T11:57:25.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:25 smithi032 ceph-mon[66455]: pgmap v220: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 592 op/s; 565 KiB/s, 33 keys/s, 25 objects/s recovering 2024-03-24T11:57:25.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:25 smithi146 ceph-mon[53008]: pgmap v220: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 592 op/s; 565 KiB/s, 33 keys/s, 25 objects/s recovering 2024-03-24T11:57:25.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 256289 9.87 MB/sec execute 511 sec latency 22.667 ms 2024-03-24T11:57:26.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 256713 9.88 MB/sec execute 512 sec latency 60.928 ms 2024-03-24T11:57:26.885 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:26 smithi032 ceph-mon[66455]: pgmap v221: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 463 op/s; 458 KiB/s, 0 keys/s, 18 objects/s recovering 2024-03-24T11:57:26.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:26.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:26.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:26.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:26 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:26.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:26 smithi032 ceph-mon[66455]: Upgrade: osd.5 is safe to restart 2024-03-24T11:57:26.967 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:26 smithi146 ceph-mon[53008]: pgmap v221: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 463 op/s; 458 KiB/s, 0 keys/s, 18 objects/s recovering 2024-03-24T11:57:26.967 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:26.968 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:26.968 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:26.968 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:26 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:57:26.968 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:26 smithi146 ceph-mon[53008]: Upgrade: osd.5 is safe to restart 2024-03-24T11:57:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 256950 9.87 MB/sec execute 513 sec latency 83.298 ms 2024-03-24T11:57:28.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 257220 9.86 MB/sec execute 514 sec latency 102.320 ms 2024-03-24T11:57:29.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 257512 9.85 MB/sec execute 515 sec latency 92.744 ms 2024-03-24T11:57:29.525 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:57:29.735 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:29 smithi032 ceph-mon[66455]: pgmap v222: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 475 op/s; 30 KiB/s, 4 objects/s recovering 2024-03-24T11:57:29.735 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:29 smithi032 ceph-mon[66455]: Upgrade: Updating osd.5 2024-03-24T11:57:29.735 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:29.735 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:57:29.735 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:29.736 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:29 smithi032 ceph-mon[66455]: Deploying daemon osd.5 on smithi146 2024-03-24T11:57:29.810 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:29 smithi146 ceph-mon[53008]: pgmap v222: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 475 op/s; 30 KiB/s, 4 objects/s recovering 2024-03-24T11:57:29.810 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:29 smithi146 ceph-mon[53008]: Upgrade: Updating osd.5 2024-03-24T11:57:29.811 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:29.811 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:57:29.811 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:29.811 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:29 smithi146 ceph-mon[53008]: Deploying daemon osd.5 on smithi146 2024-03-24T11:57:30.430 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (8m) 65s ago 17m 22.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (4m) 65s ago 16m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (4m) 25s ago 16m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 65s ago 16m 52.0M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (11m) 65s ago 11m 40.8M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (11m) 25s ago 11m 17.2M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (7m) 65s ago 18m 507M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (6m) 25s ago 16m 450M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (5m) 65s ago 18m 50.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (5m) 25s ago 16m 40.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (10m) 65s ago 10m 279M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (10m) 25s ago 10m 88.9M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 65s ago 16m 16.1M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:57:30.431 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (7m) 25s ago 16m 18.3M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 65s ago 15m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (2m) 65s ago 15m 285M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 65s ago 14m 217M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (76s) 65s ago 14m 14.9M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (29s) 25s ago 13m 12.4M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (13m) 25s ago 13m 583M 2704M 17.2.0 e1d6a67b021e 47a66faa8378 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (13m) 25s ago 13m 520M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (12m) 25s ago 12m 567M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:57:30.432 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 65s ago 15m 68.8M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:57:30.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 257739 9.84 MB/sec execute 516 sec latency 65.302 ms 2024-03-24T11:57:30.755 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:30 smithi032 ceph-mon[66455]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:30.865 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:30 smithi146 ceph-mon[53008]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:30.967 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:57:30.967 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:57:30.967 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:57:30.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:57:30.967 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:57:30.967 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 9 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:57:30.968 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "11/23 daemons upgraded", 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:57:31.395 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:57:31.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 257992 9.83 MB/sec execute 517 sec latency 63.934 ms 2024-03-24T11:57:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:31 smithi032 ceph-mon[66455]: pgmap v223: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.0 MiB/s wr, 575 op/s; 27 KiB/s, 3 objects/s recovering 2024-03-24T11:57:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:31 smithi032 ceph-mon[66455]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:31 smithi032 ceph-mon[66455]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:31 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:31 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/140721413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:31.805 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:31 smithi146 ceph-mon[53008]: pgmap v223: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.0 MiB/s wr, 575 op/s; 27 KiB/s, 3 objects/s recovering 2024-03-24T11:57:31.805 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:31 smithi146 ceph-mon[53008]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:31.805 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:31 smithi146 ceph-mon[53008]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:31.805 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:31 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:31.805 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:31 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/140721413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:31.921 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:57:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 258260 9.83 MB/sec execute 518 sec latency 63.778 ms 2024-03-24T11:57:32.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:32 smithi032 ceph-mon[66455]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:32.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:32 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3561554402' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:57:32.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:32 smithi146 systemd[1]: Stopping Ceph osd.5 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:57:32.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:32 smithi146 ceph-mon[53008]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:57:32.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:32 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3561554402' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:57:33.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:32 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[34191]: 2024-03-24T11:57:32.909+0000 7ff7145fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:57:33.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:32 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[34191]: 2024-03-24T11:57:32.909+0000 7ff7145fb700 -1 osd.5 125 *** Got signal Terminated *** 2024-03-24T11:57:33.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:32 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[34191]: 2024-03-24T11:57:32.909+0000 7ff7145fb700 -1 osd.5 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:57:33.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 258543 9.83 MB/sec execute 519 sec latency 40.613 ms 2024-03-24T11:57:33.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:33 smithi032 ceph-mon[66455]: pgmap v224: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.7 MiB/s wr, 602 op/s 2024-03-24T11:57:33.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:33 smithi032 ceph-mon[66455]: osd.5 marked itself down and dead 2024-03-24T11:57:33.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:33 smithi146 ceph-mon[53008]: pgmap v224: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.7 MiB/s wr, 602 op/s 2024-03-24T11:57:33.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:33 smithi146 ceph-mon[53008]: osd.5 marked itself down and dead 2024-03-24T11:57:34.494 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:34 smithi146 podman[59760]: 2024-03-24 11:57:34.221341989 +0000 UTC m=+1.459907726 container died 47a66faa8378b9bea6a062fb838331c0e492b37812b8f693f0fa42034eb6f749 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., RELEASE=HEAD, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, architecture=x86_64, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , ceph=True, vcs-type=git) 2024-03-24T11:57:34.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 258552 9.82 MB/sec execute 520 sec latency 975.612 ms 2024-03-24T11:57:34.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:57:34.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: osdmap e126: 8 total, 7 up, 8 in 2024-03-24T11:57:34.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.826 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.826 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:34 smithi032 ceph-mon[66455]: osdmap e127: 8 total, 7 up, 8 in 2024-03-24T11:57:34.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:34 smithi146 podman[59760]: 2024-03-24 11:57:34.601567875 +0000 UTC m=+1.840133623 container cleanup 47a66faa8378b9bea6a062fb838331c0e492b37812b8f693f0fa42034eb6f749 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, io.openshift.expose-services=, distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, version=8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., name=centos-stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-24T11:57:34.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:34 smithi146 bash[59760]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5 2024-03-24T11:57:34.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:57:34.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: osdmap e126: 8 total, 7 up, 8 in 2024-03-24T11:57:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: failed to encode map e126 with expected crc 2024-03-24T11:57:34.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:34 smithi146 ceph-mon[53008]: osdmap e127: 8 total, 7 up, 8 in 2024-03-24T11:57:35.291 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:34 smithi146 podman[59778]: 2024-03-24 11:57:34.899121674 +0000 UTC m=+0.681369548 container remove 47a66faa8378b9bea6a062fb838331c0e492b37812b8f693f0fa42034eb6f749 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.buildah.version=1.19.8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vendor=Red Hat, Inc.) 2024-03-24T11:57:35.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 258965 9.82 MB/sec execute 521 sec latency 1069.129 ms 2024-03-24T11:57:35.575 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:35 smithi146 podman[59869]: 2024-03-24 11:57:35.197252959 +0000 UTC m=+0.022330982 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:57:35.575 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:35 smithi146 podman[59869]: 2024-03-24 11:57:35.321125077 +0000 UTC m=+0.146203125 container create 58657e018bbec9272729e81272f0854960acebe0253d033049bd283ed7a9b9fe (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T11:57:35.581 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:35 smithi032 ceph-mon[66455]: pgmap v226: 97 pgs: 12 stale+active+clean, 85 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.2 MiB/s wr, 501 op/s 2024-03-24T11:57:35.581 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:35 smithi032 ceph-mon[66455]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.581 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:35 smithi032 ceph-mon[66455]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.581 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:35 smithi032 ceph-mon[66455]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:35 smithi032 ceph-mon[66455]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:35 smithi032 ceph-mon[66455]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.582 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:35 smithi032 ceph-mon[66455]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:35 smithi146 ceph-mon[53008]: pgmap v226: 97 pgs: 12 stale+active+clean, 85 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.2 MiB/s wr, 501 op/s 2024-03-24T11:57:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:35 smithi146 ceph-mon[53008]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:35 smithi146 ceph-mon[53008]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:35 smithi146 ceph-mon[53008]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:35 smithi146 ceph-mon[53008]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:35 smithi146 ceph-mon[53008]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:35 smithi146 ceph-mon[53008]: failed to encode map e127 with expected crc 2024-03-24T11:57:35.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:35 smithi146 podman[59869]: 2024-03-24 11:57:35.607936468 +0000 UTC m=+0.433014482 container init 58657e018bbec9272729e81272f0854960acebe0253d033049bd283ed7a9b9fe (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T11:57:35.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:35 smithi146 podman[59869]: 2024-03-24 11:57:35.611341296 +0000 UTC m=+0.436419310 container start 58657e018bbec9272729e81272f0854960acebe0253d033049bd283ed7a9b9fe (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T11:57:35.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:35 smithi146 podman[59869]: 2024-03-24 11:57:35.715988663 +0000 UTC m=+0.541066669 container attach 58657e018bbec9272729e81272f0854960acebe0253d033049bd283ed7a9b9fe (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T11:57:36.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:35 smithi146 podman[59869]: 2024-03-24 11:57:35.974477271 +0000 UTC m=+0.799555274 container died 58657e018bbec9272729e81272f0854960acebe0253d033049bd283ed7a9b9fe (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=) 2024-03-24T11:57:36.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 259329 9.81 MB/sec execute 522 sec latency 39.347 ms 2024-03-24T11:57:36.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:36 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 23/1554 objects degraded (1.480%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:36.846 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:36 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 23/1554 objects degraded (1.480%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:36.847 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:36 smithi146 podman[59895]: 2024-03-24 11:57:36.604073225 +0000 UTC m=+0.623307477 container remove 58657e018bbec9272729e81272f0854960acebe0253d033049bd283ed7a9b9fe (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T11:57:36.847 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:36 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-24T11:57:36.847 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:36 smithi146 systemd[1]: Stopped Ceph osd.5 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:57:36.847 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:36 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.5.service: Consumed 5min 28.493s CPU time. 2024-03-24T11:57:36.847 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:36 smithi146 systemd[1]: Starting Ceph osd.5 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:57:37.134 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:37 smithi146 podman[60001]: 2024-03-24 11:57:36.978132125 +0000 UTC m=+0.021945928 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:57:37.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 259646 9.81 MB/sec execute 523 sec latency 62.143 ms 2024-03-24T11:57:37.878 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:37 smithi032 ceph-mon[66455]: pgmap v228: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 2 peering, 8 stale+active+clean, 82 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 482 op/s; 23/1554 objects degraded (1.480%) 2024-03-24T11:57:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:37 smithi146 ceph-mon[53008]: pgmap v228: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 2 peering, 8 stale+active+clean, 82 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 482 op/s; 23/1554 objects degraded (1.480%) 2024-03-24T11:57:37.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:37 smithi146 podman[60001]: 2024-03-24 11:57:37.817225556 +0000 UTC m=+0.861039361 container create 401946a44e12d84c8ee20e46d9e7b45db7b9aa8a97fc461f738db9eda3f096d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:57:38.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 podman[60001]: 2024-03-24 11:57:38.092648618 +0000 UTC m=+1.136462444 container init 401946a44e12d84c8ee20e46d9e7b45db7b9aa8a97fc461f738db9eda3f096d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:57:38.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 podman[60001]: 2024-03-24 11:57:38.096309608 +0000 UTC m=+1.140123419 container start 401946a44e12d84c8ee20e46d9e7b45db7b9aa8a97fc461f738db9eda3f096d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T11:57:38.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 podman[60001]: 2024-03-24 11:57:38.192504927 +0000 UTC m=+1.236318736 container attach 401946a44e12d84c8ee20e46d9e7b45db7b9aa8a97fc461f738db9eda3f096d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-24T11:57:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: 5 260004 9.82 MB/sec execute 524 sec latency 38.216 ms 2024-03-24T11:57:38.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate[60016]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:57:38.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate[60016]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:57:38.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 bash[60001]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:57:38.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 bash[60001]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:57:38.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 bash[60001]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 bash[60001]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 bash[60001]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 bash[60001]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 bash[60001]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate[60016]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate[60016]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate[60016]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate[60016]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate[60016]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T11:57:38.885 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:38 smithi146 podman[60001]: 2024-03-24 11:57:38.627220861 +0000 UTC m=+1.671034674 container died 401946a44e12d84c8ee20e46d9e7b45db7b9aa8a97fc461f738db9eda3f096d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T11:57:39.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 260350 9.82 MB/sec execute 525 sec latency 48.024 ms 2024-03-24T11:57:39.633 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:39 smithi146 podman[60148]: 2024-03-24 11:57:39.320051654 +0000 UTC m=+0.686177673 container remove 401946a44e12d84c8ee20e46d9e7b45db7b9aa8a97fc461f738db9eda3f096d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T11:57:39.633 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:39 smithi146 podman[60207]: 2024-03-24 11:57:39.44273862 +0000 UTC m=+0.019955300 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:57:39.633 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:39 smithi146 podman[60207]: 2024-03-24 11:57:39.562912182 +0000 UTC m=+0.140128843 container create 3ce00834b2066ee47191ad813771ad98ad0097a962f8a64ef1a51bd4c8d3699f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:57:39.653 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:39 smithi032 ceph-mon[66455]: pgmap v229: 97 pgs: 7 active+undersized, 14 active+undersized+degraded, 2 peering, 3 stale+active+clean, 71 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.6 MiB/s wr, 443 op/s; 111/1584 objects degraded (7.008%); 258 B/s, 0 objects/s recovering 2024-03-24T11:57:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:39 smithi146 ceph-mon[53008]: pgmap v229: 97 pgs: 7 active+undersized, 14 active+undersized+degraded, 2 peering, 3 stale+active+clean, 71 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.6 MiB/s wr, 443 op/s; 111/1584 objects degraded (7.008%); 258 B/s, 0 objects/s recovering 2024-03-24T11:57:39.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:39 smithi146 podman[60207]: 2024-03-24 11:57:39.807330683 +0000 UTC m=+0.384547345 container init 3ce00834b2066ee47191ad813771ad98ad0097a962f8a64ef1a51bd4c8d3699f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T11:57:39.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:39 smithi146 podman[60207]: 2024-03-24 11:57:39.811694857 +0000 UTC m=+0.388911518 container start 3ce00834b2066ee47191ad813771ad98ad0097a962f8a64ef1a51bd4c8d3699f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T11:57:40.353 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:39 smithi146 bash[60207]: 3ce00834b2066ee47191ad813771ad98ad0097a962f8a64ef1a51bd4c8d3699f 2024-03-24T11:57:40.354 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:39 smithi146 systemd[1]: Started Ceph osd.5 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:57:40.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 260613 9.81 MB/sec execute 526 sec latency 53.713 ms 2024-03-24T11:57:40.637 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:40 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T11:57:40.353+0000 7f63f459a7c0 -1 Falling back to public interface 2024-03-24T11:57:41.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:40 smithi032 ceph-mon[66455]: pgmap v230: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 3.6 MiB/s wr, 289 op/s; 189/1698 objects degraded (11.131%); 258 B/s, 0 objects/s recovering 2024-03-24T11:57:41.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:41.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:40 smithi146 ceph-mon[53008]: pgmap v230: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 3.6 MiB/s wr, 289 op/s; 189/1698 objects degraded (11.131%); 258 B/s, 0 objects/s recovering 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:41.194 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:41.195 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:41.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 260921 9.80 MB/sec execute 527 sec latency 46.206 ms 2024-03-24T11:57:42.114 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:41 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 186/1668 objects degraded (11.151%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:42.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:41 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 186/1668 objects degraded (11.151%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 261315 9.80 MB/sec execute 528 sec latency 56.150 ms 2024-03-24T11:57:43.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:42 smithi032 ceph-mon[66455]: pgmap v231: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 478 op/s; 186/1668 objects degraded (11.151%); 254 B/s, 0 objects/s recovering 2024-03-24T11:57:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:42 smithi146 ceph-mon[53008]: pgmap v231: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 478 op/s; 186/1668 objects degraded (11.151%); 254 B/s, 0 objects/s recovering 2024-03-24T11:57:43.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 261752 9.80 MB/sec execute 529 sec latency 38.359 ms 2024-03-24T11:57:44.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 262134 9.81 MB/sec execute 530 sec latency 37.465 ms 2024-03-24T11:57:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 262491 9.81 MB/sec execute 531 sec latency 65.641 ms 2024-03-24T11:57:45.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:45 smithi032 ceph-mon[66455]: pgmap v232: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.8 MiB/s wr, 504 op/s; 192/1683 objects degraded (11.408%); 206 B/s, 0 objects/s recovering 2024-03-24T11:57:45.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:45.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:45.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:45 smithi146 ceph-mon[53008]: pgmap v232: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.8 MiB/s wr, 504 op/s; 192/1683 objects degraded (11.408%); 206 B/s, 0 objects/s recovering 2024-03-24T11:57:45.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:45.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:46.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 262811 9.81 MB/sec execute 532 sec latency 38.319 ms 2024-03-24T11:57:46.807 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:46 smithi032 ceph-mon[66455]: pgmap v233: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.6 MiB/s wr, 471 op/s; 193/1689 objects degraded (11.427%); 185 B/s, 0 objects/s recovering 2024-03-24T11:57:46.807 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:46 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:46.807 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:46 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:46 smithi146 ceph-mon[53008]: pgmap v233: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.6 MiB/s wr, 471 op/s; 193/1689 objects degraded (11.427%); 185 B/s, 0 objects/s recovering 2024-03-24T11:57:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:46 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:47.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:46 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:47.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 263055 9.80 MB/sec execute 533 sec latency 58.079 ms 2024-03-24T11:57:47.809 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:47 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 193/1689 objects degraded (11.427%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:47.825 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:47 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 193/1689 objects degraded (11.427%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:48.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 263340 9.80 MB/sec execute 534 sec latency 57.837 ms 2024-03-24T11:57:48.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:48 smithi032 ceph-mon[66455]: pgmap v234: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.9 MiB/s wr, 501 op/s; 188/1683 objects degraded (11.171%); 172 B/s, 0 objects/s recovering 2024-03-24T11:57:49.135 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:48 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T11:57:48.753+0000 7f63f459a7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-24T11:57:49.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:48 smithi146 ceph-mon[53008]: pgmap v234: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.9 MiB/s wr, 501 op/s; 188/1683 objects degraded (11.171%); 172 B/s, 0 objects/s recovering 2024-03-24T11:57:49.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 263567 9.79 MB/sec execute 535 sec latency 53.437 ms 2024-03-24T11:57:50.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 263863 9.80 MB/sec execute 536 sec latency 57.849 ms 2024-03-24T11:57:50.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T11:57:50.341+0000 7f63f459a7c0 -1 osd.5 125 log_to_monitors true 2024-03-24T11:57:50.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: pgmap v235: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.1 MiB/s wr, 446 op/s; 187/1668 objects degraded (11.211%) 2024-03-24T11:57:50.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:50.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-24T11:57:50.894 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='osd.5 [v2:172.21.15.146:6808/3776867261,v1:172.21.15.146:6809/3776867261]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:57:50.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:50 smithi032 ceph-mon[66455]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:57:51.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: pgmap v235: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.1 MiB/s wr, 446 op/s; 187/1668 objects degraded (11.211%) 2024-03-24T11:57:51.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='osd.5 [v2:172.21.15.146:6808/3776867261,v1:172.21.15.146:6809/3776867261]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:57:51.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:50 smithi146 ceph-mon[53008]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:57:51.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 264039 9.78 MB/sec execute 537 sec latency 171.673 ms 2024-03-24T11:57:51.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T11:57:51.264+0000 7f63e19ea700 -1 osd.5 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:57:51.929 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:51 smithi032 ceph-mon[66455]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:57:51.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:51 smithi032 ceph-mon[66455]: osdmap e128: 8 total, 7 up, 8 in 2024-03-24T11:57:51.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:51 smithi032 ceph-mon[66455]: from='osd.5 [v2:172.21.15.146:6808/3776867261,v1:172.21.15.146:6809/3776867261]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:51.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:51 smithi032 ceph-mon[66455]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:51.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:51 smithi032 ceph-mon[66455]: failed to encode map e128 with expected crc 2024-03-24T11:57:51.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:51 smithi032 ceph-mon[66455]: failed to encode map e128 with expected crc 2024-03-24T11:57:51.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:51 smithi032 ceph-mon[66455]: from='osd.5 ' entity='osd.5' 2024-03-24T11:57:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-mon[53008]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:57:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-mon[53008]: osdmap e128: 8 total, 7 up, 8 in 2024-03-24T11:57:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-mon[53008]: from='osd.5 [v2:172.21.15.146:6808/3776867261,v1:172.21.15.146:6809/3776867261]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-mon[53008]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:57:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-mon[53008]: failed to encode map e128 with expected crc 2024-03-24T11:57:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-mon[53008]: failed to encode map e128 with expected crc 2024-03-24T11:57:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:51 smithi146 ceph-mon[53008]: from='osd.5 ' entity='osd.5' 2024-03-24T11:57:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 264351 9.78 MB/sec execute 538 sec latency 205.780 ms 2024-03-24T11:57:52.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:52 smithi032 ceph-mon[66455]: pgmap v237: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 487 op/s; 179/1656 objects degraded (10.809%) 2024-03-24T11:57:52.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:52 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:57:52.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:52 smithi032 ceph-mon[66455]: osd.5 [v2:172.21.15.146:6808/3776867261,v1:172.21.15.146:6809/3776867261] boot 2024-03-24T11:57:52.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:52 smithi032 ceph-mon[66455]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:57:52.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:52 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:57:52.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:52 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 179/1656 objects degraded (10.809%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:52.949 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:52 smithi032 ceph-mon[66455]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:57:53.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:52 smithi146 ceph-mon[53008]: pgmap v237: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 487 op/s; 179/1656 objects degraded (10.809%) 2024-03-24T11:57:53.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:52 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:57:53.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:52 smithi146 ceph-mon[53008]: osd.5 [v2:172.21.15.146:6808/3776867261,v1:172.21.15.146:6809/3776867261] boot 2024-03-24T11:57:53.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:52 smithi146 ceph-mon[53008]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:57:53.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:52 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:57:53.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:52 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 179/1656 objects degraded (10.809%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:53.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:52 smithi146 ceph-mon[53008]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:57:53.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 264531 9.76 MB/sec execute 539 sec latency 1205.814 ms 2024-03-24T11:57:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e129 with expected crc 2024-03-24T11:57:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e129 with expected crc 2024-03-24T11:57:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: failed to encode map e130 with expected crc 2024-03-24T11:57:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:53 smithi032 ceph-mon[66455]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:57:54.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e129 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e129 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: failed to encode map e130 with expected crc 2024-03-24T11:57:54.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:53 smithi146 ceph-mon[53008]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:57:54.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 264759 9.75 MB/sec execute 540 sec latency 1295.103 ms 2024-03-24T11:57:54.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:54 smithi032 ceph-mon[66455]: failed to encode map e131 with expected crc 2024-03-24T11:57:54.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:54 smithi032 ceph-mon[66455]: failed to encode map e131 with expected crc 2024-03-24T11:57:54.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:54 smithi032 ceph-mon[66455]: pgmap v241: 97 pgs: 1 remapped+peering, 7 active+undersized, 19 active+undersized+degraded, 70 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 508 op/s; 142/1620 objects degraded (8.765%); 17 KiB/s, 6 objects/s recovering 2024-03-24T11:57:54.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:54 smithi032 ceph-mon[66455]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:57:55.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:54 smithi146 ceph-mon[53008]: failed to encode map e131 with expected crc 2024-03-24T11:57:55.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:54 smithi146 ceph-mon[53008]: failed to encode map e131 with expected crc 2024-03-24T11:57:55.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:54 smithi146 ceph-mon[53008]: pgmap v241: 97 pgs: 1 remapped+peering, 7 active+undersized, 19 active+undersized+degraded, 70 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 508 op/s; 142/1620 objects degraded (8.765%); 17 KiB/s, 6 objects/s recovering 2024-03-24T11:57:55.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:54 smithi146 ceph-mon[53008]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:57:55.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 265005 9.75 MB/sec execute 541 sec latency 853.257 ms 2024-03-24T11:57:56.026 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.026 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.026 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.026 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.027 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.027 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.027 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:57:56.027 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:56.027 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: failed to encode map e132 with expected crc 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:57:56.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:57:56.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:56.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 265283 9.75 MB/sec execute 542 sec latency 932.054 ms 2024-03-24T11:57:56.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: failed to encode map e133 with expected crc 2024-03-24T11:57:56.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: failed to encode map e133 with expected crc 2024-03-24T11:57:56.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: failed to encode map e133 with expected crc 2024-03-24T11:57:56.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: failed to encode map e133 with expected crc 2024-03-24T11:57:56.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: failed to encode map e133 with expected crc 2024-03-24T11:57:56.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: failed to encode map e133 with expected crc 2024-03-24T11:57:56.796 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: pgmap v244: 97 pgs: 1 remapped+peering, 5 active+undersized, 1 active+undersized+remapped, 17 active+undersized+degraded, 73 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s wr, 93 op/s; 136/1599 objects degraded (8.505%); 1.2 MiB/s, 8 objects/s recovering 2024-03-24T11:57:56.797 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:56 smithi032 ceph-mon[66455]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:57:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: failed to encode map e133 with expected crc 2024-03-24T11:57:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: failed to encode map e133 with expected crc 2024-03-24T11:57:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: failed to encode map e133 with expected crc 2024-03-24T11:57:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: failed to encode map e133 with expected crc 2024-03-24T11:57:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: failed to encode map e133 with expected crc 2024-03-24T11:57:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: failed to encode map e133 with expected crc 2024-03-24T11:57:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: pgmap v244: 97 pgs: 1 remapped+peering, 5 active+undersized, 1 active+undersized+remapped, 17 active+undersized+degraded, 73 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.9 MiB/s wr, 93 op/s; 136/1599 objects degraded (8.505%); 1.2 MiB/s, 8 objects/s recovering 2024-03-24T11:57:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:56 smithi146 ceph-mon[53008]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:57:57.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 265674 9.75 MB/sec execute 543 sec latency 977.028 ms 2024-03-24T11:57:57.811 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:57 smithi032 ceph-mon[66455]: failed to encode map e134 with expected crc 2024-03-24T11:57:57.811 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:57 smithi032 ceph-mon[66455]: failed to encode map e134 with expected crc 2024-03-24T11:57:57.811 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:57 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 136/1599 objects degraded (8.505%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:58.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:57 smithi146 ceph-mon[53008]: failed to encode map e134 with expected crc 2024-03-24T11:57:58.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:57 smithi146 ceph-mon[53008]: failed to encode map e134 with expected crc 2024-03-24T11:57:58.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:57 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 136/1599 objects degraded (8.505%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T11:57:58.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 266062 9.75 MB/sec execute 544 sec latency 44.263 ms 2024-03-24T11:57:58.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:58 smithi032 ceph-mon[66455]: pgmap v246: 97 pgs: 1 peering, 1 remapped+peering, 95 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.5 MiB/s wr, 352 op/s; 2.5 MiB/s, 38 objects/s recovering 2024-03-24T11:57:58.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:58 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 136/1599 objects degraded (8.505%), 17 pgs degraded) 2024-03-24T11:57:58.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:57:58 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:57:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:58 smithi146 ceph-mon[53008]: pgmap v246: 97 pgs: 1 peering, 1 remapped+peering, 95 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.5 MiB/s wr, 352 op/s; 2.5 MiB/s, 38 objects/s recovering 2024-03-24T11:57:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:58 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 136/1599 objects degraded (8.505%), 17 pgs degraded) 2024-03-24T11:57:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:57:58 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:57:59.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 266367 9.75 MB/sec execute 545 sec latency 35.548 ms 2024-03-24T11:58:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 266720 9.75 MB/sec execute 546 sec latency 38.998 ms 2024-03-24T11:58:00.858 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:00 smithi032 ceph-mon[66455]: pgmap v247: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.8 MiB/s wr, 309 op/s; 1.9 MiB/s, 31 objects/s recovering 2024-03-24T11:58:01.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:00 smithi146 ceph-mon[53008]: pgmap v247: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.8 MiB/s wr, 309 op/s; 1.9 MiB/s, 31 objects/s recovering 2024-03-24T11:58:01.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 267058 9.75 MB/sec execute 547 sec latency 52.171 ms 2024-03-24T11:58:02.463 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:58:02.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 267341 9.75 MB/sec execute 548 sec latency 40.045 ms 2024-03-24T11:58:02.936 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:02 smithi032 ceph-mon[66455]: pgmap v248: 97 pgs: 97 active+clean; 345 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.9 MiB/s wr, 513 op/s; 1.1 MiB/s, 25 objects/s recovering 2024-03-24T11:58:02.936 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:02 smithi032 ceph-mon[66455]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:02 smithi146 ceph-mon[53008]: pgmap v248: 97 pgs: 97 active+clean; 345 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.9 MiB/s wr, 513 op/s; 1.1 MiB/s, 25 objects/s recovering 2024-03-24T11:58:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:02 smithi146 ceph-mon[53008]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:03.481 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:58:03.481 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (8m) 98s ago 17m 22.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:58:03.481 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (4m) 98s ago 17m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:58:03.481 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (4m) 19s ago 16m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (8m) 98s ago 17m 52.0M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (11m) 98s ago 11m 40.8M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (11m) 19s ago 11m 17.3M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (7m) 98s ago 18m 507M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (6m) 19s ago 16m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 98s ago 18m 50.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (5m) 19s ago 16m 42.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (11m) 98s ago 11m 279M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (11m) 19s ago 11m 89.0M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 98s ago 17m 16.1M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (7m) 19s ago 16m 18.2M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (4m) 98s ago 16m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 98s ago 15m 285M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:58:03.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 98s ago 15m 217M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:58:03.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (109s) 98s ago 14m 14.9M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T11:58:03.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (62s) 19s ago 14m 280M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T11:58:03.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (24s) 19s ago 13m 12.6M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T11:58:03.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (13m) 19s ago 13m 555M 2704M 17.2.0 e1d6a67b021e 0e944340d070 2024-03-24T11:58:03.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (13m) 19s ago 13m 603M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:58:03.483 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 98s ago 16m 68.8M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:58:03.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 267613 9.75 MB/sec execute 549 sec latency 61.397 ms 2024-03-24T11:58:03.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:03 smithi032 ceph-mon[66455]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:03.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:03 smithi032 ceph-mon[66455]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:04.071 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 6 2024-03-24T11:58:04.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 10 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:58:04.073 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:58:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:03 smithi146 ceph-mon[53008]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:03 smithi146 ceph-mon[53008]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:04.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 267904 9.74 MB/sec execute 550 sec latency 61.895 ms 2024-03-24T11:58:04.555 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:58:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:58:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:58:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "12/23 daemons upgraded", 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:58:04.556 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:58:04.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:04 smithi032 ceph-mon[66455]: pgmap v249: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.1 MiB/s wr, 471 op/s; 953 KiB/s, 22 objects/s recovering 2024-03-24T11:58:04.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:04 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3654478940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:04.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:04 smithi032 ceph-mon[66455]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:04 smithi146 ceph-mon[53008]: pgmap v249: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.1 MiB/s wr, 471 op/s; 953 KiB/s, 22 objects/s recovering 2024-03-24T11:58:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:04 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3654478940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:04 smithi146 ceph-mon[53008]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:05.267 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:58:05.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 268207 9.73 MB/sec execute 551 sec latency 48.936 ms 2024-03-24T11:58:05.915 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:05 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:58:05.915 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:05 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:58:05.915 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:05 smithi146 ceph-mon[53008]: Upgrade: osd.6 is safe to restart 2024-03-24T11:58:05.915 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:05 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/2341972737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:58:05.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:05 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:58:05.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:05 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:58:05.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:05 smithi032 ceph-mon[66455]: Upgrade: osd.6 is safe to restart 2024-03-24T11:58:05.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:05 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2341972737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:58:06.497 INFO:teuthology.orchestra.run.smithi032.stdout: 5 268474 9.72 MB/sec execute 552 sec latency 46.448 ms 2024-03-24T11:58:07.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:06 smithi032 ceph-mon[66455]: pgmap v250: 97 pgs: 97 active+clean; 346 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.8 MiB/s wr, 496 op/s; 781 KiB/s, 18 objects/s recovering 2024-03-24T11:58:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:06 smithi146 ceph-mon[53008]: pgmap v250: 97 pgs: 97 active+clean; 346 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.8 MiB/s wr, 496 op/s; 781 KiB/s, 18 objects/s recovering 2024-03-24T11:58:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 268858 9.73 MB/sec execute 553 sec latency 72.920 ms 2024-03-24T11:58:08.017 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:07 smithi032 ceph-mon[66455]: Upgrade: Updating osd.6 2024-03-24T11:58:08.017 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:08.017 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:58:08.017 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:08.017 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:07 smithi032 ceph-mon[66455]: Deploying daemon osd.6 on smithi146 2024-03-24T11:58:08.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:07 smithi146 ceph-mon[53008]: Upgrade: Updating osd.6 2024-03-24T11:58:08.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:08.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:58:08.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:08.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:07 smithi146 ceph-mon[53008]: Deploying daemon osd.6 on smithi146 2024-03-24T11:58:08.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 269278 9.73 MB/sec execute 554 sec latency 53.434 ms 2024-03-24T11:58:08.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:08 smithi032 ceph-mon[66455]: pgmap v251: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 378 op/s; 479 KiB/s, 6 objects/s recovering 2024-03-24T11:58:08.939 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:08 smithi146 ceph-mon[53008]: pgmap v251: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 378 op/s; 479 KiB/s, 6 objects/s recovering 2024-03-24T11:58:09.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 269618 9.73 MB/sec execute 555 sec latency 53.140 ms 2024-03-24T11:58:10.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 269872 9.73 MB/sec execute 556 sec latency 59.222 ms 2024-03-24T11:58:10.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:10 smithi032 ceph-mon[66455]: pgmap v252: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.8 MiB/s wr, 455 op/s; 97 KiB/s, 2 objects/s recovering 2024-03-24T11:58:10.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:10.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:10.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:11.124 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:10 smithi146 ceph-mon[53008]: pgmap v252: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.8 MiB/s wr, 455 op/s; 97 KiB/s, 2 objects/s recovering 2024-03-24T11:58:11.125 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:11.125 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:11.125 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:11.384 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:11 smithi146 systemd[1]: Stopping Ceph osd.6 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:58:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 270277 9.74 MB/sec execute 557 sec latency 52.363 ms 2024-03-24T11:58:11.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:11 smithi032 ceph-mon[66455]: osd.6 marked itself down and dead 2024-03-24T11:58:12.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:11 smithi146 ceph-mon[53008]: osd.6 marked itself down and dead 2024-03-24T11:58:12.135 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[37180]: 2024-03-24T11:58:11.771+0000 7fb8b28de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:58:12.135 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[37180]: 2024-03-24T11:58:11.771+0000 7fb8b28de700 -1 osd.6 134 *** Got signal Terminated *** 2024-03-24T11:58:12.135 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[37180]: 2024-03-24T11:58:11.771+0000 7fb8b28de700 -1 osd.6 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:58:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 270522 9.73 MB/sec execute 558 sec latency 549.495 ms 2024-03-24T11:58:12.820 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:12 smithi146 podman[62042]: 2024-03-24 11:58:12.514585937 +0000 UTC m=+0.986667840 container died 0e944340d070eba78006be30d5997eafde034cd6e881de8d09f66043e0703874 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, ceph=True, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T11:58:13.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:12 smithi032 ceph-mon[66455]: pgmap v253: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.6 MiB/s wr, 563 op/s 2024-03-24T11:58:13.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:12 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:58:13.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:12 smithi032 ceph-mon[66455]: failed to encode map e135 with expected crc 2024-03-24T11:58:13.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:12 smithi032 ceph-mon[66455]: osdmap e135: 8 total, 7 up, 8 in 2024-03-24T11:58:13.116 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:12 smithi032 ceph-mon[66455]: failed to encode map e135 with expected crc 2024-03-24T11:58:13.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:12 smithi146 ceph-mon[53008]: pgmap v253: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.6 MiB/s wr, 563 op/s 2024-03-24T11:58:13.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:12 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:58:13.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:12 smithi146 ceph-mon[53008]: failed to encode map e135 with expected crc 2024-03-24T11:58:13.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:12 smithi146 ceph-mon[53008]: osdmap e135: 8 total, 7 up, 8 in 2024-03-24T11:58:13.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:12 smithi146 ceph-mon[53008]: failed to encode map e135 with expected crc 2024-03-24T11:58:13.135 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:13 smithi146 podman[62042]: 2024-03-24 11:58:13.020949844 +0000 UTC m=+1.493031755 container cleanup 0e944340d070eba78006be30d5997eafde034cd6e881de8d09f66043e0703874 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8) 2024-03-24T11:58:13.135 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:13 smithi146 bash[62042]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6 2024-03-24T11:58:13.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 270756 9.72 MB/sec execute 559 sec latency 972.956 ms 2024-03-24T11:58:13.634 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:13 smithi146 podman[62060]: 2024-03-24 11:58:13.322901372 +0000 UTC m=+0.804492180 container remove 0e944340d070eba78006be30d5997eafde034cd6e881de8d09f66043e0703874 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, distribution-scope=public, architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-24T11:58:13.888 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:13 smithi032 ceph-mon[66455]: failed to encode map e136 with expected crc 2024-03-24T11:58:13.888 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:13 smithi032 ceph-mon[66455]: osdmap e136: 8 total, 7 up, 8 in 2024-03-24T11:58:13.888 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:13 smithi032 ceph-mon[66455]: failed to encode map e136 with expected crc 2024-03-24T11:58:13.888 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:13 smithi032 ceph-mon[66455]: failed to encode map e136 with expected crc 2024-03-24T11:58:14.001 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:13 smithi146 ceph-mon[53008]: failed to encode map e136 with expected crc 2024-03-24T11:58:14.001 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:13 smithi146 ceph-mon[53008]: osdmap e136: 8 total, 7 up, 8 in 2024-03-24T11:58:14.001 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:13 smithi146 ceph-mon[53008]: failed to encode map e136 with expected crc 2024-03-24T11:58:14.001 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:13 smithi146 ceph-mon[53008]: failed to encode map e136 with expected crc 2024-03-24T11:58:14.002 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:13 smithi146 podman[62149]: 2024-03-24 11:58:13.621081616 +0000 UTC m=+0.023451486 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:58:14.002 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:13 smithi146 podman[62149]: 2024-03-24 11:58:13.748941553 +0000 UTC m=+0.151311433 container create 7c32ded361cb8dd330758607460536bbfe5d4925a35a2da5afc6353cba623c18 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T11:58:14.002 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:14 smithi146 podman[62149]: 2024-03-24 11:58:14.002293536 +0000 UTC m=+0.404663429 container init 7c32ded361cb8dd330758607460536bbfe5d4925a35a2da5afc6353cba623c18 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T11:58:14.361 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:14 smithi146 podman[62149]: 2024-03-24 11:58:14.007043942 +0000 UTC m=+0.409413817 container start 7c32ded361cb8dd330758607460536bbfe5d4925a35a2da5afc6353cba623c18 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T11:58:14.362 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:14 smithi146 podman[62149]: 2024-03-24 11:58:14.127206645 +0000 UTC m=+0.529576514 container attach 7c32ded361cb8dd330758607460536bbfe5d4925a35a2da5afc6353cba623c18 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0) 2024-03-24T11:58:14.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 271064 9.72 MB/sec execute 560 sec latency 37.244 ms 2024-03-24T11:58:14.634 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:14 smithi146 podman[62149]: 2024-03-24 11:58:14.362861393 +0000 UTC m=+0.765231277 container died 7c32ded361cb8dd330758607460536bbfe5d4925a35a2da5afc6353cba623c18 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T11:58:14.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:14 smithi032 ceph-mon[66455]: pgmap v256: 97 pgs: 1 active+undersized, 5 active+undersized+degraded, 1 peering, 8 stale+active+clean, 82 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 480 op/s; 43/1674 objects degraded (2.569%) 2024-03-24T11:58:14.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:14 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 43/1674 objects degraded (2.569%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:15.113 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 podman[62176]: 2024-03-24 11:58:15.098843136 +0000 UTC m=+0.729904129 container remove 7c32ded361cb8dd330758607460536bbfe5d4925a35a2da5afc6353cba623c18 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-24T11:58:15.113 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:14 smithi146 ceph-mon[53008]: pgmap v256: 97 pgs: 1 active+undersized, 5 active+undersized+degraded, 1 peering, 8 stale+active+clean, 82 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 480 op/s; 43/1674 objects degraded (2.569%) 2024-03-24T11:58:15.113 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:14 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 43/1674 objects degraded (2.569%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:15.367 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-24T11:58:15.367 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 systemd[1]: Stopped Ceph osd.6 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:58:15.367 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.6.service: Consumed 4min 59.425s CPU time. 2024-03-24T11:58:15.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 271418 9.71 MB/sec execute 561 sec latency 50.465 ms 2024-03-24T11:58:15.632 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 systemd[1]: Starting Ceph osd.6 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:58:15.633 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 podman[62282]: 2024-03-24 11:58:15.50113089 +0000 UTC m=+0.018718577 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:58:15.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 podman[62282]: 2024-03-24 11:58:15.633730973 +0000 UTC m=+0.151318648 container create ecfe9f01eee7c4709d134965f49591a5beb032edf58435307a752df960cf173d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b) 2024-03-24T11:58:16.384 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 podman[62282]: 2024-03-24 11:58:15.900404393 +0000 UTC m=+0.417992068 container init ecfe9f01eee7c4709d134965f49591a5beb032edf58435307a752df960cf173d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:58:16.384 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:15 smithi146 podman[62282]: 2024-03-24 11:58:15.90374164 +0000 UTC m=+0.421329325 container start ecfe9f01eee7c4709d134965f49591a5beb032edf58435307a752df960cf173d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T11:58:16.384 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 podman[62282]: 2024-03-24 11:58:16.012197952 +0000 UTC m=+0.529785631 container attach ecfe9f01eee7c4709d134965f49591a5beb032edf58435307a752df960cf173d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-24T11:58:16.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 271752 9.71 MB/sec execute 562 sec latency 56.777 ms 2024-03-24T11:58:16.715 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate[62297]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate[62297]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 bash[62282]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 bash[62282]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 bash[62282]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 bash[62282]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 bash[62282]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 bash[62282]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 bash[62282]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate[62297]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate[62297]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate[62297]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-24T11:58:16.716 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate[62297]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:58:16.717 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate[62297]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-24T11:58:16.717 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:16 smithi146 podman[62282]: 2024-03-24 11:58:16.451480223 +0000 UTC m=+0.969067889 container died ecfe9f01eee7c4709d134965f49591a5beb032edf58435307a752df960cf173d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T11:58:16.938 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:16 smithi032 ceph-mon[66455]: pgmap v257: 97 pgs: 7 active+undersized, 12 active+undersized+degraded, 1 peering, 3 stale+active+clean, 74 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.4 MiB/s wr, 569 op/s; 81/1671 objects degraded (4.847%) 2024-03-24T11:58:17.073 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:16 smithi146 ceph-mon[53008]: pgmap v257: 97 pgs: 7 active+undersized, 12 active+undersized+degraded, 1 peering, 3 stale+active+clean, 74 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.4 MiB/s wr, 569 op/s; 81/1671 objects degraded (4.847%) 2024-03-24T11:58:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 272033 9.70 MB/sec execute 563 sec latency 40.721 ms 2024-03-24T11:58:17.858 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:17 smithi146 podman[62413]: 2024-03-24 11:58:17.501203518 +0000 UTC m=+1.043098555 container remove ecfe9f01eee7c4709d134965f49591a5beb032edf58435307a752df960cf173d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T11:58:17.859 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:17 smithi146 podman[62476]: 2024-03-24 11:58:17.623555818 +0000 UTC m=+0.020485987 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:58:17.859 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:17 smithi146 podman[62476]: 2024-03-24 11:58:17.743873876 +0000 UTC m=+0.140804046 container create 8bab439aede48a17390bd800de536d514ef88607f0c473d496f23b4e2e517317 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T11:58:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:17 smithi146 podman[62476]: 2024-03-24 11:58:17.971479475 +0000 UTC m=+0.368409653 container init 8bab439aede48a17390bd800de536d514ef88607f0c473d496f23b4e2e517317 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T11:58:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:17 smithi146 podman[62476]: 2024-03-24 11:58:17.974647789 +0000 UTC m=+0.371577967 container start 8bab439aede48a17390bd800de536d514ef88607f0c473d496f23b4e2e517317 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T11:58:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:17 smithi146 ceph-osd[62494]: -- 172.21.15.146:0/313343435 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0480411e0 con 0x55b048012c00 2024-03-24T11:58:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:18 smithi146 bash[62476]: 8bab439aede48a17390bd800de536d514ef88607f0c473d496f23b4e2e517317 2024-03-24T11:58:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:18 smithi146 systemd[1]: Started Ceph osd.6 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:58:18.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 272473 9.71 MB/sec execute 564 sec latency 78.713 ms 2024-03-24T11:58:18.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T11:58:18.523+0000 7f7fece097c0 -1 Falling back to public interface 2024-03-24T11:58:19.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:19 smithi032 ceph-mon[66455]: pgmap v258: 97 pgs: 10 active+undersized, 24 active+undersized+degraded, 1 peering, 62 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 422 op/s; 174/1674 objects degraded (10.394%) 2024-03-24T11:58:19.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:19.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:19.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:19.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:19 smithi146 ceph-mon[53008]: pgmap v258: 97 pgs: 10 active+undersized, 24 active+undersized+degraded, 1 peering, 62 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 422 op/s; 174/1674 objects degraded (10.394%) 2024-03-24T11:58:19.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:19.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:19.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:19.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 272907 9.71 MB/sec execute 565 sec latency 42.379 ms 2024-03-24T11:58:20.268 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:20 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 192/1665 objects degraded (11.532%), 25 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:20.298 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:20 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 192/1665 objects degraded (11.532%), 25 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:20.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 273273 9.71 MB/sec execute 566 sec latency 50.549 ms 2024-03-24T11:58:21.235 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:21 smithi146 ceph-mon[53008]: pgmap v259: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.3 MiB/s wr, 455 op/s; 192/1665 objects degraded (11.532%) 2024-03-24T11:58:21.295 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:21 smithi032 ceph-mon[66455]: pgmap v259: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.3 MiB/s wr, 455 op/s; 192/1665 objects degraded (11.532%) 2024-03-24T11:58:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 273540 9.71 MB/sec execute 567 sec latency 43.803 ms 2024-03-24T11:58:22.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 273907 9.72 MB/sec execute 568 sec latency 46.908 ms 2024-03-24T11:58:23.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:23 smithi032 ceph-mon[66455]: pgmap v260: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.9 MiB/s wr, 399 op/s; 189/1665 objects degraded (11.351%) 2024-03-24T11:58:23.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:23 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:23.359 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:23 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:23.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 274202 9.71 MB/sec execute 569 sec latency 50.806 ms 2024-03-24T11:58:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:23 smithi146 ceph-mon[53008]: pgmap v260: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.9 MiB/s wr, 399 op/s; 189/1665 objects degraded (11.351%) 2024-03-24T11:58:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:23 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:23 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:24.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 274478 9.71 MB/sec execute 570 sec latency 60.366 ms 2024-03-24T11:58:24.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:24.652 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:24.867 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:24.867 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:25.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 274770 9.71 MB/sec execute 571 sec latency 54.003 ms 2024-03-24T11:58:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:25 smithi146 ceph-mon[53008]: pgmap v261: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 352 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.2 MiB/s wr, 551 op/s; 192/1680 objects degraded (11.429%) 2024-03-24T11:58:25.634 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T11:58:25.393+0000 7f7fece097c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-24T11:58:25.665 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:25 smithi032 ceph-mon[66455]: pgmap v261: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 352 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.2 MiB/s wr, 551 op/s; 192/1680 objects degraded (11.429%) 2024-03-24T11:58:26.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 275076 9.70 MB/sec execute 572 sec latency 53.878 ms 2024-03-24T11:58:26.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:26 smithi032 ceph-mon[66455]: pgmap v262: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.8 MiB/s wr, 498 op/s; 200/1698 objects degraded (11.779%) 2024-03-24T11:58:26.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:26.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:26.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:26 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 200/1698 objects degraded (11.779%), 25 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:26.973 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T11:58:26.798+0000 7f7fece097c0 -1 osd.6 134 log_to_monitors true 2024-03-24T11:58:26.973 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:26 smithi146 ceph-mon[53008]: pgmap v262: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.8 MiB/s wr, 498 op/s; 200/1698 objects degraded (11.779%) 2024-03-24T11:58:26.974 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:26.974 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:26.974 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:26 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 200/1698 objects degraded (11.779%), 25 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 275374 9.69 MB/sec execute 573 sec latency 56.853 ms 2024-03-24T11:58:27.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:27 smithi032 ceph-mon[66455]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:58:28.133 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:27 smithi146 ceph-mon[53008]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:58:28.384 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T11:58:28.213+0000 7f7fda259700 -1 osd.6 134 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:58:28.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 275565 9.68 MB/sec execute 574 sec latency 198.948 ms 2024-03-24T11:58:28.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: pgmap v263: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.6 MiB/s wr, 430 op/s; 198/1680 objects degraded (11.786%) 2024-03-24T11:58:28.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: failed to encode map e137 with expected crc 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: osdmap e137: 8 total, 7 up, 8 in 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: failed to encode map e137 with expected crc 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: failed to encode map e137 with expected crc 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:28.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:28.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:28.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:28.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:28.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:28 smithi032 ceph-mon[66455]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:58:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: pgmap v263: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.6 MiB/s wr, 430 op/s; 198/1680 objects degraded (11.786%) 2024-03-24T11:58:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:58:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: failed to encode map e137 with expected crc 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: osdmap e137: 8 total, 7 up, 8 in 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: failed to encode map e137 with expected crc 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: failed to encode map e137 with expected crc 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207]' entity='osd.6' 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:29.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:29.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:28 smithi146 ceph-mon[53008]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:58:29.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 275911 9.69 MB/sec execute 575 sec latency 283.491 ms 2024-03-24T11:58:30.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:29 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:58:30.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:29 smithi146 ceph-mon[53008]: osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207] boot 2024-03-24T11:58:30.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:29 smithi146 ceph-mon[53008]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:58:30.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:58:30.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:29 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:58:30.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:29 smithi032 ceph-mon[66455]: osd.6 [v2:172.21.15.146:6816/3446325207,v1:172.21.15.146:6817/3446325207] boot 2024-03-24T11:58:30.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:29 smithi032 ceph-mon[66455]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:58:30.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:58:30.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 275911 9.67 MB/sec execute 576 sec latency 1283.550 ms 2024-03-24T11:58:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:30 smithi146 ceph-mon[53008]: failed to encode map e138 with expected crc 2024-03-24T11:58:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:30 smithi146 ceph-mon[53008]: failed to encode map e138 with expected crc 2024-03-24T11:58:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:30 smithi146 ceph-mon[53008]: failed to encode map e138 with expected crc 2024-03-24T11:58:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:30 smithi146 ceph-mon[53008]: pgmap v266: 97 pgs: 8 peering, 4 active+undersized, 23 active+undersized+degraded, 62 active+clean; 346 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.2 MiB/s wr, 526 op/s; 178/1659 objects degraded (10.729%) 2024-03-24T11:58:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:30 smithi146 ceph-mon[53008]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:58:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:30 smithi032 ceph-mon[66455]: failed to encode map e138 with expected crc 2024-03-24T11:58:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:30 smithi032 ceph-mon[66455]: failed to encode map e138 with expected crc 2024-03-24T11:58:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:30 smithi032 ceph-mon[66455]: failed to encode map e138 with expected crc 2024-03-24T11:58:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:30 smithi032 ceph-mon[66455]: pgmap v266: 97 pgs: 8 peering, 4 active+undersized, 23 active+undersized+degraded, 62 active+clean; 346 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.2 MiB/s wr, 526 op/s; 178/1659 objects degraded (10.729%) 2024-03-24T11:58:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:30 smithi032 ceph-mon[66455]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:58:31.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 275911 9.65 MB/sec execute 577 sec latency 2283.595 ms 2024-03-24T11:58:31.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:31 smithi032 ceph-mon[66455]: failed to encode map e139 with expected crc 2024-03-24T11:58:31.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:31 smithi032 ceph-mon[66455]: failed to encode map e139 with expected crc 2024-03-24T11:58:31.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:31 smithi032 ceph-mon[66455]: failed to encode map e139 with expected crc 2024-03-24T11:58:31.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:31 smithi032 ceph-mon[66455]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:58:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:31 smithi146 ceph-mon[53008]: failed to encode map e139 with expected crc 2024-03-24T11:58:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:31 smithi146 ceph-mon[53008]: failed to encode map e139 with expected crc 2024-03-24T11:58:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:31 smithi146 ceph-mon[53008]: failed to encode map e139 with expected crc 2024-03-24T11:58:32.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:31 smithi146 ceph-mon[53008]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:58:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 276227 9.65 MB/sec execute 578 sec latency 2503.022 ms 2024-03-24T11:58:32.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:32 smithi032 ceph-mon[66455]: failed to encode map e140 with expected crc 2024-03-24T11:58:32.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:32 smithi032 ceph-mon[66455]: failed to encode map e140 with expected crc 2024-03-24T11:58:32.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:32 smithi032 ceph-mon[66455]: pgmap v269: 97 pgs: 2 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 8 peering, 2 active+undersized, 13 active+undersized+degraded, 70 active+clean; 343 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 5.0 MiB/s wr, 628 op/s; 134/1572 objects degraded (8.524%); 52 KiB/s, 57 keys/s, 9 objects/s recovering 2024-03-24T11:58:32.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:32 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 134/1572 objects degraded (8.524%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:32.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:32 smithi032 ceph-mon[66455]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:58:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:32 smithi146 ceph-mon[53008]: failed to encode map e140 with expected crc 2024-03-24T11:58:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:32 smithi146 ceph-mon[53008]: failed to encode map e140 with expected crc 2024-03-24T11:58:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:32 smithi146 ceph-mon[53008]: pgmap v269: 97 pgs: 2 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 8 peering, 2 active+undersized, 13 active+undersized+degraded, 70 active+clean; 343 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 5.0 MiB/s wr, 628 op/s; 134/1572 objects degraded (8.524%); 52 KiB/s, 57 keys/s, 9 objects/s recovering 2024-03-24T11:58:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:32 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 134/1572 objects degraded (8.524%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:33.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:32 smithi146 ceph-mon[53008]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:58:33.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 276239 9.63 MB/sec execute 579 sec latency 997.317 ms 2024-03-24T11:58:33.902 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:33 smithi032 ceph-mon[66455]: failed to encode map e141 with expected crc 2024-03-24T11:58:33.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:33 smithi032 ceph-mon[66455]: failed to encode map e141 with expected crc 2024-03-24T11:58:33.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:33 smithi032 ceph-mon[66455]: failed to encode map e141 with expected crc 2024-03-24T11:58:33.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:33 smithi032 ceph-mon[66455]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:58:34.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:33 smithi146 ceph-mon[53008]: failed to encode map e141 with expected crc 2024-03-24T11:58:34.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:33 smithi146 ceph-mon[53008]: failed to encode map e141 with expected crc 2024-03-24T11:58:34.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:33 smithi146 ceph-mon[53008]: failed to encode map e141 with expected crc 2024-03-24T11:58:34.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:33 smithi146 ceph-mon[53008]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:58:34.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 276639 9.63 MB/sec execute 580 sec latency 1042.859 ms 2024-03-24T11:58:34.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:34 smithi032 ceph-mon[66455]: failed to encode map e142 with expected crc 2024-03-24T11:58:34.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:34 smithi032 ceph-mon[66455]: failed to encode map e142 with expected crc 2024-03-24T11:58:34.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:34 smithi032 ceph-mon[66455]: pgmap v272: 97 pgs: 2 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 12 peering, 81 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.9 MiB/s wr, 279 op/s; 39/1518 objects degraded (2.569%); 4.6 MiB/s, 57 keys/s, 31 objects/s recovering 2024-03-24T11:58:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:34 smithi146 ceph-mon[53008]: failed to encode map e142 with expected crc 2024-03-24T11:58:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:34 smithi146 ceph-mon[53008]: failed to encode map e142 with expected crc 2024-03-24T11:58:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:34 smithi146 ceph-mon[53008]: pgmap v272: 97 pgs: 2 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 12 peering, 81 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.9 MiB/s wr, 279 op/s; 39/1518 objects degraded (2.569%); 4.6 MiB/s, 57 keys/s, 31 objects/s recovering 2024-03-24T11:58:35.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 276968 9.64 MB/sec execute 581 sec latency 29.092 ms 2024-03-24T11:58:35.808 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:58:36.498 INFO:teuthology.orchestra.run.smithi032.stdout: 5 277341 9.64 MB/sec execute 582 sec latency 38.692 ms 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (9m) 2m ago 18m 22.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 2m ago 18m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (5m) 14s ago 17m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (8m) 2m ago 17m 52.0M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (12m) 2m ago 12m 40.8M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (12m) 14s ago 12m 17.4M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (8m) 2m ago 19m 507M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (7m) 14s ago 17m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:58:36.882 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 2m ago 19m 50.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (6m) 14s ago 17m 42.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (11m) 2m ago 11m 279M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (11m) 14s ago 11m 91.1M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 2m ago 17m 16.1M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (8m) 14s ago 17m 17.4M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (4m) 2m ago 16m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 2m ago 16m 285M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (3m) 2m ago 15m 217M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (2m) 2m ago 15m 14.9M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (96s) 14s ago 14m 300M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (57s) 14s ago 14m 277M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (19s) 14s ago 14m 12.4M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (13m) 14s ago 13m 658M 2704M 17.2.0 e1d6a67b021e 7369fa9f802b 2024-03-24T11:58:36.883 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 2m ago 16m 68.8M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:58:36.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:36 smithi032 ceph-mon[66455]: pgmap v273: 97 pgs: 2 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 4 peering, 89 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 212 op/s; 39/1533 objects degraded (2.544%); 3.4 MiB/s, 41 keys/s, 29 objects/s recovering 2024-03-24T11:58:36.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:36 smithi032 ceph-mon[66455]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:36.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:36 smithi032 ceph-mon[66455]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:37.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:36 smithi146 ceph-mon[53008]: pgmap v273: 97 pgs: 2 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 4 peering, 89 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 212 op/s; 39/1533 objects degraded (2.544%); 3.4 MiB/s, 41 keys/s, 29 objects/s recovering 2024-03-24T11:58:37.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:36 smithi146 ceph-mon[53008]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:37.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:36 smithi146 ceph-mon[53008]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:37.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 277666 9.64 MB/sec execute 583 sec latency 55.070 ms 2024-03-24T11:58:37.520 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:58:37.520 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:58:37.520 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 7 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:58:37.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-24T11:58:37.522 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 11 2024-03-24T11:58:37.522 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:58:37.522 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:58:37.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:37 smithi032 ceph-mon[66455]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:37.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:37 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 39/1533 objects degraded (2.544%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:37.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:37 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/145658331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:38.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:37 smithi146 ceph-mon[53008]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:38.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:37 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 39/1533 objects degraded (2.544%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:38.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:37 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/145658331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:58:38.155 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:58:38.155 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:58:38.155 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:58:38.155 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "13/23 daemons upgraded", 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:58:38.156 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:58:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 278045 9.64 MB/sec execute 584 sec latency 44.505 ms 2024-03-24T11:58:38.768 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T11:58:38.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:38 smithi032 ceph-mon[66455]: pgmap v274: 97 pgs: 4 peering, 93 active+clean; 345 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.9 MiB/s wr, 277 op/s; 3.3 MiB/s, 31 objects/s recovering 2024-03-24T11:58:38.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:38 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1533 objects degraded (2.544%), 3 pgs degraded) 2024-03-24T11:58:38.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:38 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:58:38.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:38 smithi032 ceph-mon[66455]: from='client.44363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:38 smithi146 ceph-mon[53008]: pgmap v274: 97 pgs: 4 peering, 93 active+clean; 345 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.9 MiB/s wr, 277 op/s; 3.3 MiB/s, 31 objects/s recovering 2024-03-24T11:58:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:38 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1533 objects degraded (2.544%), 3 pgs degraded) 2024-03-24T11:58:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:38 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:58:39.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:38 smithi146 ceph-mon[53008]: from='client.44363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:58:39.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 278378 9.64 MB/sec execute 585 sec latency 53.086 ms 2024-03-24T11:58:39.749 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:39 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/4276174972' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:58:40.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:39 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/4276174972' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:58:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 278772 9.63 MB/sec execute 586 sec latency 40.617 ms 2024-03-24T11:58:40.756 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:40 smithi032 ceph-mon[66455]: pgmap v275: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.1 MiB/s wr, 419 op/s; 2.6 MiB/s, 23 objects/s recovering 2024-03-24T11:58:40.756 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:40.756 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:40 smithi146 ceph-mon[53008]: pgmap v275: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.1 MiB/s wr, 419 op/s; 2.6 MiB/s, 23 objects/s recovering 2024-03-24T11:58:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:41.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 278985 9.62 MB/sec execute 587 sec latency 51.596 ms 2024-03-24T11:58:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 279303 9.63 MB/sec execute 588 sec latency 54.669 ms 2024-03-24T11:58:42.877 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:42 smithi032 ceph-mon[66455]: pgmap v276: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 406 op/s; 2.3 MiB/s, 20 objects/s recovering 2024-03-24T11:58:43.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:42 smithi146 ceph-mon[53008]: pgmap v276: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 406 op/s; 2.3 MiB/s, 20 objects/s recovering 2024-03-24T11:58:43.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 279650 9.63 MB/sec execute 589 sec latency 68.158 ms 2024-03-24T11:58:43.900 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:43.900 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:43 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:43.900 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:43 smithi032 ceph-mon[66455]: Upgrade: osd.7 is safe to restart 2024-03-24T11:58:44.055 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:44.055 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:43 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:58:44.055 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:43 smithi146 ceph-mon[53008]: Upgrade: osd.7 is safe to restart 2024-03-24T11:58:44.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 279987 9.63 MB/sec execute 590 sec latency 69.790 ms 2024-03-24T11:58:44.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:44 smithi032 ceph-mon[66455]: pgmap v277: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 400 op/s; 288 KiB/s, 9 objects/s recovering 2024-03-24T11:58:45.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:44 smithi146 ceph-mon[53008]: pgmap v277: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 400 op/s; 288 KiB/s, 9 objects/s recovering 2024-03-24T11:58:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 280332 9.62 MB/sec execute 591 sec latency 51.069 ms 2024-03-24T11:58:46.214 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:45 smithi032 ceph-mon[66455]: Upgrade: Updating osd.7 2024-03-24T11:58:46.214 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:46.214 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:58:46.214 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:46.214 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:45 smithi032 ceph-mon[66455]: Deploying daemon osd.7 on smithi146 2024-03-24T11:58:46.214 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:45 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:45 smithi146 ceph-mon[53008]: Upgrade: Updating osd.7 2024-03-24T11:58:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:58:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:45 smithi146 ceph-mon[53008]: Deploying daemon osd.7 on smithi146 2024-03-24T11:58:46.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:45 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:46.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 280568 9.62 MB/sec execute 592 sec latency 36.057 ms 2024-03-24T11:58:46.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:46 smithi032 ceph-mon[66455]: pgmap v278: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.9 MiB/s wr, 369 op/s; 245 KiB/s, 8 objects/s recovering 2024-03-24T11:58:46.986 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:46 smithi146 ceph-mon[53008]: pgmap v278: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.9 MiB/s wr, 369 op/s; 245 KiB/s, 8 objects/s recovering 2024-03-24T11:58:47.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 280931 9.62 MB/sec execute 593 sec latency 40.037 ms 2024-03-24T11:58:48.502 INFO:teuthology.orchestra.run.smithi032.stdout: 5 281220 9.62 MB/sec execute 594 sec latency 49.609 ms 2024-03-24T11:58:49.050 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:48 smithi032 ceph-mon[66455]: pgmap v279: 97 pgs: 97 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 483 op/s; 242 KiB/s, 5 objects/s recovering 2024-03-24T11:58:49.131 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:48 smithi146 systemd[1]: Stopping Ceph osd.7 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:58:49.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:48 smithi146 ceph-mon[53008]: pgmap v279: 97 pgs: 97 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 483 op/s; 242 KiB/s, 5 objects/s recovering 2024-03-24T11:58:49.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:49 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[40263]: 2024-03-24T11:58:49.140+0000 7facc5728700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:58:49.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:49 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[40263]: 2024-03-24T11:58:49.140+0000 7facc5728700 -1 osd.7 142 *** Got signal Terminated *** 2024-03-24T11:58:49.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:49 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[40263]: 2024-03-24T11:58:49.140+0000 7facc5728700 -1 osd.7 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:58:49.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 281486 9.62 MB/sec execute 595 sec latency 59.509 ms 2024-03-24T11:58:50.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:49 smithi032 ceph-mon[66455]: osd.7 marked itself down and dead 2024-03-24T11:58:50.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:49 smithi146 ceph-mon[53008]: osd.7 marked itself down and dead 2024-03-24T11:58:50.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 281601 9.61 MB/sec execute 596 sec latency 523.494 ms 2024-03-24T11:58:50.975 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:50 smithi146 podman[64274]: 2024-03-24 11:58:50.694924435 +0000 UTC m=+1.725117190 container died 7369fa9f802b57f0a9398b14c73b36ad5764fc9971bb3810904523a78a57780d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, distribution-scope=public, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_CLEAN=True, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T11:58:51.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:50 smithi032 ceph-mon[66455]: pgmap v280: 97 pgs: 97 active+clean; 348 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-03-24T11:58:51.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:50 smithi032 ceph-mon[66455]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:58:51.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:50 smithi032 ceph-mon[66455]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-24T11:58:51.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:50 smithi032 ceph-mon[66455]: osdmap e143: 8 total, 7 up, 8 in 2024-03-24T11:58:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:50 smithi146 ceph-mon[53008]: pgmap v280: 97 pgs: 97 active+clean; 348 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-03-24T11:58:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:50 smithi146 ceph-mon[53008]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:58:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:50 smithi146 ceph-mon[53008]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-24T11:58:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:50 smithi146 ceph-mon[53008]: osdmap e143: 8 total, 7 up, 8 in 2024-03-24T11:58:51.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:51 smithi146 podman[64274]: 2024-03-24 11:58:51.142399433 +0000 UTC m=+2.172592199 container cleanup 7369fa9f802b57f0a9398b14c73b36ad5764fc9971bb3810904523a78a57780d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, ceph=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vendor=Red Hat, Inc., vcs-type=git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-24T11:58:51.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:51 smithi146 bash[64274]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7 2024-03-24T11:58:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 281751 9.61 MB/sec execute 597 sec latency 1058.120 ms 2024-03-24T11:58:51.876 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:51 smithi146 podman[64321]: 2024-03-24 11:58:51.485687934 +0000 UTC m=+0.792027386 container remove 7369fa9f802b57f0a9398b14c73b36ad5764fc9971bb3810904523a78a57780d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, version=8, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, maintainer=Guillaume Abrioux , architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-24T11:58:51.995 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:51 smithi032 ceph-mon[66455]: osdmap e144: 8 total, 7 up, 8 in 2024-03-24T11:58:52.134 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:51 smithi146 podman[64406]: 2024-03-24 11:58:51.779688635 +0000 UTC m=+0.020636515 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:58:52.134 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:51 smithi146 podman[64406]: 2024-03-24 11:58:51.903318697 +0000 UTC m=+0.144266576 container create 671cb5c54fc6f82e5d0b77ea96bcf5ae060de6616f4b8403e1a26eefd4f3256c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5) 2024-03-24T11:58:52.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:51 smithi146 ceph-mon[53008]: osdmap e144: 8 total, 7 up, 8 in 2024-03-24T11:58:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 282090 9.59 MB/sec execute 598 sec latency 74.771 ms 2024-03-24T11:58:52.538 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:52 smithi146 podman[64406]: 2024-03-24 11:58:52.173182827 +0000 UTC m=+0.414130708 container init 671cb5c54fc6f82e5d0b77ea96bcf5ae060de6616f4b8403e1a26eefd4f3256c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:58:52.538 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:52 smithi146 podman[64406]: 2024-03-24 11:58:52.176707333 +0000 UTC m=+0.417655222 container start 671cb5c54fc6f82e5d0b77ea96bcf5ae060de6616f4b8403e1a26eefd4f3256c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T11:58:52.538 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:52 smithi146 podman[64406]: 2024-03-24 11:58:52.264902876 +0000 UTC m=+0.505850770 container attach 671cb5c54fc6f82e5d0b77ea96bcf5ae060de6616f4b8403e1a26eefd4f3256c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T11:58:52.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:52 smithi146 podman[64406]: 2024-03-24 11:58:52.539304282 +0000 UTC m=+0.780252179 container died 671cb5c54fc6f82e5d0b77ea96bcf5ae060de6616f4b8403e1a26eefd4f3256c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-24T11:58:53.016 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:52 smithi032 ceph-mon[66455]: pgmap v283: 97 pgs: 12 peering, 10 stale+active+clean, 75 active+clean; 350 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-03-24T11:58:53.185 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:52 smithi146 ceph-mon[53008]: pgmap v283: 97 pgs: 12 peering, 10 stale+active+clean, 75 active+clean; 350 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-03-24T11:58:53.448 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 podman[64431]: 2024-03-24 11:58:53.18660998 +0000 UTC m=+0.636846850 container remove 671cb5c54fc6f82e5d0b77ea96bcf5ae060de6616f4b8403e1a26eefd4f3256c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T11:58:53.448 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-24T11:58:53.448 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 systemd[1]: Stopped Ceph osd.7 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:58:53.448 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 systemd[1]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.7.service: Consumed 7min 53.810s CPU time. 2024-03-24T11:58:53.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 282440 9.59 MB/sec execute 599 sec latency 36.224 ms 2024-03-24T11:58:53.737 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 systemd[1]: Starting Ceph osd.7 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:58:53.737 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 podman[64528]: 2024-03-24 11:58:53.59598328 +0000 UTC m=+0.022436170 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:58:53.737 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 podman[64528]: 2024-03-24 11:58:53.738060522 +0000 UTC m=+0.164513404 container create 292baaa301ffa3f37fb671a92fabe869e0d84a5d784e35e8c874309c40c13c58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:58:54.304 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:54 smithi032 ceph-mon[66455]: Health check failed: Degraded data redundancy: 96/1686 objects degraded (5.694%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:54.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-mon[53008]: Health check failed: Degraded data redundancy: 96/1686 objects degraded (5.694%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T11:58:54.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 podman[64528]: 2024-03-24 11:58:53.991231992 +0000 UTC m=+0.417684879 container init 292baaa301ffa3f37fb671a92fabe869e0d84a5d784e35e8c874309c40c13c58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:58:54.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:53 smithi146 podman[64528]: 2024-03-24 11:58:53.995003132 +0000 UTC m=+0.421456018 container start 292baaa301ffa3f37fb671a92fabe869e0d84a5d784e35e8c874309c40c13c58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T11:58:54.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 podman[64528]: 2024-03-24 11:58:54.099510248 +0000 UTC m=+0.525963140 container attach 292baaa301ffa3f37fb671a92fabe869e0d84a5d784e35e8c874309c40c13c58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:58:54.499 INFO:teuthology.orchestra.run.smithi032.stdout: 5 cleanup 600 sec 2024-03-24T11:58:54.752 INFO:teuthology.orchestra.run.smithi032.stdout: 0 cleanup 600 sec 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Operation Count AvgLat MaxLat 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: ---------------------------------------- 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: NTCreateX 184233 4.687 2490.963 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Close 135235 0.141 7.826 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Rename 7820 9.868 2343.868 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Unlink 37297 7.295 2327.257 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Qpathinfo 167037 1.177 1039.744 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Qfileinfo 29132 0.008 0.771 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Qfsinfo 30693 0.923 1957.133 2024-03-24T11:58:54.753 INFO:teuthology.orchestra.run.smithi032.stdout: Sfileinfo 15037 13.521 2503.014 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: Find 64628 1.012 27.459 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: WriteX 91111 13.913 2491.192 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: ReadX 289440 0.010 3.176 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: LockX 602 0.412 6.317 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: UnlockX 602 0.366 9.574 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: Flush 12929 0.002 0.043 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:58:54.754 INFO:teuthology.orchestra.run.smithi032.stdout:Throughput 9.58631 MB/sec 5 clients 5 procs max_latency=2503.022 ms 2024-03-24T11:58:54.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-03-24T11:58:54.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-03-24T11:58:54.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate[64542]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate[64542]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 bash[64528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 bash[64528]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 bash[64528]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 bash[64528]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 bash[64528]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 bash[64528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 bash[64528]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate[64542]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate[64542]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate[64542]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate[64542]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate[64542]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-24T11:58:54.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:54 smithi146 podman[64528]: 2024-03-24 11:58:54.587350078 +0000 UTC m=+1.013802963 container died 292baaa301ffa3f37fb671a92fabe869e0d84a5d784e35e8c874309c40c13c58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-24T11:58:54.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-03-24T11:58:54.948 INFO:teuthology.orchestra.run.smithi032.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-24T11:58:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:58:54.948 INFO:teuthology.orchestra.run.smithi032.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-03-24T11:58:54.949 INFO:teuthology.orchestra.run.smithi032.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-24T11:58:55.049 INFO:teuthology.orchestra.run.smithi032.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-24T11:58:55.049 INFO:teuthology.orchestra.run.smithi032.stdout:releasing clients 2024-03-24T11:58:55.130 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:55 smithi032 ceph-mon[66455]: pgmap v284: 97 pgs: 5 active+undersized, 15 active+undersized+degraded, 12 peering, 5 stale+active+clean, 60 active+clean; 350 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.3 MiB/s wr, 569 op/s; 96/1686 objects degraded (5.694%) 2024-03-24T11:58:55.236 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:55 smithi146 ceph-mon[53008]: pgmap v284: 97 pgs: 5 active+undersized, 15 active+undersized+degraded, 12 peering, 5 stale+active+clean, 60 active+clean; 350 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.3 MiB/s wr, 569 op/s; 96/1686 objects degraded (5.694%) 2024-03-24T11:58:55.597 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:55 smithi146 podman[64669]: 2024-03-24 11:58:55.237190635 +0000 UTC m=+0.642947234 container remove 292baaa301ffa3f37fb671a92fabe869e0d84a5d784e35e8c874309c40c13c58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-activate, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-24T11:58:55.597 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:55 smithi146 podman[64729]: 2024-03-24 11:58:55.359171687 +0000 UTC m=+0.020888571 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:58:55.597 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:55 smithi146 podman[64729]: 2024-03-24 11:58:55.479877611 +0000 UTC m=+0.141594485 container create 85b15c0b61146be19720f32b7640204ab4e3ba8695b07e37d9a7ff6f5749ceb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T11:58:56.758 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:56.758 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:56.758 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:56 smithi032 ceph-mon[66455]: pgmap v285: 97 pgs: 5 active+undersized, 25 active+undersized+degraded, 12 peering, 55 active+clean; 347 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 419 op/s; 195/1674 objects degraded (11.649%) 2024-03-24T11:58:56.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:56.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:56.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:56 smithi146 ceph-mon[53008]: pgmap v285: 97 pgs: 5 active+undersized, 25 active+undersized+degraded, 12 peering, 55 active+clean; 347 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 419 op/s; 195/1674 objects degraded (11.649%) 2024-03-24T11:58:56.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:56 smithi146 podman[64729]: 2024-03-24 11:58:56.624590801 +0000 UTC m=+1.286307666 container init 85b15c0b61146be19720f32b7640204ab4e3ba8695b07e37d9a7ff6f5749ceb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T11:58:56.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:56 smithi146 podman[64729]: 2024-03-24 11:58:56.627801567 +0000 UTC m=+1.289518430 container start 85b15c0b61146be19720f32b7640204ab4e3ba8695b07e37d9a7ff6f5749ceb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T11:58:56.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:56 smithi146 ceph-osd[64747]: -- 172.21.15.146:0/1479977338 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55607fb751e0 con 0x55607fb47000 2024-03-24T11:58:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: 5 323 26.01 MB/sec execute 1 sec latency 254.037 ms 2024-03-24T11:58:57.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:57 smithi146 bash[64729]: 85b15c0b61146be19720f32b7640204ab4e3ba8695b07e37d9a7ff6f5749ceb1 2024-03-24T11:58:57.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:57 smithi146 systemd[1]: Started Ceph osd.7 for 0ddfea52-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:58:57.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:58:57 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T11:58:57.173+0000 7f23f00f47c0 -1 Falling back to public interface 2024-03-24T11:58:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: 5 460 23.93 MB/sec execute 2 sec latency 44.745 ms 2024-03-24T11:58:58.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:58 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:58.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:58 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:58.329 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:58 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:58.368 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:58 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:58.368 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:58 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:58:58.369 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:58 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:59.049 INFO:teuthology.orchestra.run.smithi032.stdout: 5 732 24.59 MB/sec execute 3 sec latency 26.775 ms 2024-03-24T11:58:59.133 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:58:59 smithi032 ceph-mon[66455]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 331 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s wr, 276 op/s; 245/1521 objects degraded (16.108%) 2024-03-24T11:58:59.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:58:59 smithi146 ceph-mon[53008]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 331 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s wr, 276 op/s; 245/1521 objects degraded (16.108%) 2024-03-24T11:59:00.049 INFO:teuthology.orchestra.run.smithi032.stdout: 5 1074 20.47 MB/sec execute 4 sec latency 33.575 ms 2024-03-24T11:59:00.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:00 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 219/1167 objects degraded (18.766%), 35 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:00.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:00 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 219/1167 objects degraded (18.766%), 35 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:01.049 INFO:teuthology.orchestra.run.smithi032.stdout: 5 cleanup 5 sec 2024-03-24T11:59:01.167 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:01 smithi032 ceph-mon[66455]: pgmap v287: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 317 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 470 op/s; 219/1167 objects degraded (18.766%) 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: 0 cleanup 5 sec 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Operation Count AvgLat MaxLat 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: ---------------------------------------- 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: NTCreateX 987 7.228 52.198 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Close 874 6.718 62.715 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Rename 50 11.156 33.567 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Unlink 97 3.698 21.499 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Qpathinfo 584 1.415 20.617 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Qfileinfo 406 15.545 44.072 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Qfsinfo 165 0.414 2.279 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Sfileinfo 170 13.314 23.272 2024-03-24T11:59:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Find 258 0.788 2.527 2024-03-24T11:59:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: WriteX 1861 0.025 0.070 2024-03-24T11:59:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: ReadX 1174 0.013 1.043 2024-03-24T11:59:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: Flush 123 12.919 26.795 2024-03-24T11:59:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T11:59:01.272 INFO:teuthology.orchestra.run.smithi032.stdout:Throughput 20.4677 MB/sec 5 clients 5 procs max_latency=254.037 ms 2024-03-24T11:59:01.273 DEBUG:teuthology.parallel:result is None 2024-03-24T11:59:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:01 smithi146 ceph-mon[53008]: pgmap v287: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 317 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 470 op/s; 219/1167 objects degraded (18.766%) 2024-03-24T11:59:02.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:02 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:02.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:02 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:02.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:02.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:03.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:03 smithi146 ceph-mon[53008]: pgmap v288: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 329 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 12 MiB/s wr, 440 op/s; 220/1293 objects degraded (17.015%) 2024-03-24T11:59:03.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:03.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:03.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:03 smithi032 ceph-mon[66455]: pgmap v288: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 329 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 12 MiB/s wr, 440 op/s; 220/1293 objects degraded (17.015%) 2024-03-24T11:59:03.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:03.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:05.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:05 smithi146 ceph-mon[53008]: pgmap v289: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 12 MiB/s wr, 495 op/s; 233/1404 objects degraded (16.595%) 2024-03-24T11:59:05.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:05 smithi032 ceph-mon[66455]: pgmap v289: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 12 MiB/s wr, 495 op/s; 233/1404 objects degraded (16.595%) 2024-03-24T11:59:06.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:59:06 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T11:59:06.061+0000 7f23f00f47c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-24T11:59:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: pgmap v290: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 320 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 11 MiB/s wr, 380 op/s; 170/1194 objects degraded (14.238%) 2024-03-24T11:59:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all osd 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-24T11:59:07.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-24T11:59:07.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: Upgrade: Setting require_osd_release to 19 squid 2024-03-24T11:59:07.387 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-24T11:59:07.387 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 170/1194 objects degraded (14.238%), 35 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:07.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: pgmap v290: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 320 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 11 MiB/s wr, 380 op/s; 170/1194 objects degraded (14.238%) 2024-03-24T11:59:07.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all osd 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-24T11:59:07.453 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-24T11:59:07.454 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: Upgrade: Setting require_osd_release to 19 squid 2024-03-24T11:59:07.455 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-24T11:59:07.455 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:07 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 170/1194 objects degraded (14.238%), 35 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:07.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:59:07 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T11:59:07.590+0000 7f23f00f47c0 -1 osd.7 142 log_to_monitors true 2024-03-24T11:59:08.384 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T11:59:08.279+0000 7f23e154c700 -1 osd.7 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:59:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-mon[53008]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-24T11:59:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-24T11:59:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-mon[53008]: osdmap e145: 8 total, 7 up, 8 in 2024-03-24T11:59:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi032.qzdcjy"]}]: dispatch 2024-03-24T11:59:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-mon[53008]: Upgrade: It appears safe to stop mds.foofs.smithi032.qzdcjy 2024-03-24T11:59:08.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-mon[53008]: from='osd.7 [v2:172.21.15.146:6824/116010560,v1:172.21.15.146:6825/116010560]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:59:08.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:08 smithi146 ceph-mon[53008]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:59:08.433 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:08 smithi032 ceph-mon[66455]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-24T11:59:08.433 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-24T11:59:08.433 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:08 smithi032 ceph-mon[66455]: osdmap e145: 8 total, 7 up, 8 in 2024-03-24T11:59:08.433 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi032.qzdcjy"]}]: dispatch 2024-03-24T11:59:08.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:08 smithi032 ceph-mon[66455]: Upgrade: It appears safe to stop mds.foofs.smithi032.qzdcjy 2024-03-24T11:59:08.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:08 smithi032 ceph-mon[66455]: from='osd.7 [v2:172.21.15.146:6824/116010560,v1:172.21.15.146:6825/116010560]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:59:08.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:08 smithi032 ceph-mon[66455]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:59:09.130 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:59:09.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:09 smithi032 ceph-mon[66455]: pgmap v292: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 290 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 11 MiB/s wr, 418 op/s; 150/921 objects degraded (16.287%) 2024-03-24T11:59:09.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:09 smithi032 ceph-mon[66455]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:59:09.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:09 smithi032 ceph-mon[66455]: osdmap e146: 8 total, 7 up, 8 in 2024-03-24T11:59:09.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:09 smithi032 ceph-mon[66455]: from='osd.7 [v2:172.21.15.146:6824/116010560,v1:172.21.15.146:6825/116010560]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:59:09.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:09 smithi032 ceph-mon[66455]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:59:09.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:09 smithi146 ceph-mon[53008]: pgmap v292: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 290 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 11 MiB/s wr, 418 op/s; 150/921 objects degraded (16.287%) 2024-03-24T11:59:09.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:09 smithi146 ceph-mon[53008]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:59:09.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:09 smithi146 ceph-mon[53008]: osdmap e146: 8 total, 7 up, 8 in 2024-03-24T11:59:09.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:09 smithi146 ceph-mon[53008]: from='osd.7 [v2:172.21.15.146:6824/116010560,v1:172.21.15.146:6825/116010560]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:59:09.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:09 smithi146 ceph-mon[53008]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-24T11:59:09.822 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:59:09.822 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (9m) 2m ago 18m 22.0M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:59:09.822 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 2m ago 18m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:59:09.822 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (5m) 8s ago 18m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:59:09.822 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (9m) 2m ago 18m 52.0M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:59:09.822 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (12m) 2m ago 12m 40.8M - 17.2.0 e1d6a67b021e 6f16a39a45d3 2024-03-24T11:59:09.823 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (12m) 8s ago 12m 17.5M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:59:09.823 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (8m) 2m ago 19m 507M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:59:09.823 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (7m) 8s ago 18m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:59:09.823 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (7m) 2m ago 20m 50.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:59:09.823 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (6m) 8s ago 17m 43.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:59:09.823 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (12m) 2m ago 12m 279M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:59:09.824 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (12m) 8s ago 12m 91.1M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:59:09.824 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (9m) 2m ago 18m 16.1M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:59:09.824 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (8m) 8s ago 17m 17.6M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:59:09.824 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (5m) 2m ago 17m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:59:09.824 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (4m) 2m ago 16m 285M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:59:09.824 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (3m) 2m ago 16m 217M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:59:09.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (2m) 2m ago 15m 14.9M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T11:59:09.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (2m) 8s ago 15m 340M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T11:59:09.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (90s) 8s ago 15m 297M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T11:59:09.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (52s) 8s ago 14m 276M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T11:59:09.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (14s) 8s ago 14m 14.8M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 85b15c0b6114 2024-03-24T11:59:09.825 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 2m ago 17m 68.8M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:59:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:59:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:59:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-24T11:59:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 12 2024-03-24T11:59:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:59:10.253 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:59:10.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:10.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:59:10.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: osd.7 [v2:172.21.15.146:6824/116010560,v1:172.21.15.146:6825/116010560] boot 2024-03-24T11:59:10.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:59:10.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:59:10.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:10.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:10.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:59:10.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:10.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3560568603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:10.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:10 smithi032 ceph-mon[66455]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:59:10.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:10.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:59:10.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: osd.7 [v2:172.21.15.146:6824/116010560,v1:172.21.15.146:6825/116010560] boot 2024-03-24T11:59:10.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:59:10.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:59:10.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:10.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:10.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi032.qzdcjy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:59:10.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:10.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3560568603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:10.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:10 smithi146 ceph-mon[53008]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:59:10.637 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:59:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:59:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:59:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:59:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "14/23 daemons upgraded", 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:59:10.638 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:59:11.078 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN Degraded data redundancy: 87/540 objects degraded (16.111%), 30 pgs degraded 2024-03-24T11:59:11.078 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 87/540 objects degraded (16.111%), 30 pgs degraded 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-24T11:59:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-24T11:59:11.080 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-03-24T11:59:11.081 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-24T11:59:11.081 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-03-24T11:59:11.081 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-24T11:59:11.081 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-03-24T11:59:11.081 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-24T11:59:11.081 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-24T11:59:11.081 INFO:teuthology.orchestra.run.smithi032.stdout: pg 4.a is active+undersized+degraded, acting [6,1] 2024-03-24T11:59:11.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: pgmap v295: 97 pgs: 4 peering, 8 active+undersized, 30 active+undersized+degraded, 55 active+clean; 251 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 337 KiB/s wr, 65 op/s; 87/540 objects degraded (16.111%) 2024-03-24T11:59:11.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:11.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: Upgrade: Updating mds.foofs.smithi032.qzdcjy 2024-03-24T11:59:11.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: Deploying daemon mds.foofs.smithi032.qzdcjy on smithi032 2024-03-24T11:59:11.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:11.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:11.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:11.577 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/386500732' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:59:11.577 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:11 smithi032 ceph-mon[66455]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: pgmap v295: 97 pgs: 4 peering, 8 active+undersized, 30 active+undersized+degraded, 55 active+clean; 251 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 337 KiB/s wr, 65 op/s; 87/540 objects degraded (16.111%) 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: Upgrade: Updating mds.foofs.smithi032.qzdcjy 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: Deploying daemon mds.foofs.smithi032.qzdcjy on smithi032 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:11.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/386500732' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:59:11.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:11 smithi146 ceph-mon[53008]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:59:12.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T11:59:12.349+0000 7f63c91b9700 -1 bdev(0x564372f10000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-24T11:59:12.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-mon[53008]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T11:59:12.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-mon[53008]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:59:12.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-mon[53008]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:59:12.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-mon[53008]: Standby daemon mds.foofs.smithi146.xsrsue assigned to filesystem foofs as rank 0 2024-03-24T11:59:12.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-mon[53008]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T11:59:12.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-mon[53008]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:59:12.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:12 smithi146 ceph-mon[53008]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[66451]: 2024-03-24T11:59:12.272+0000 7efe750aa700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-mon[66455]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-mon[66455]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-mon[66455]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-mon[66455]: Standby daemon mds.foofs.smithi146.xsrsue assigned to filesystem foofs as rank 0 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-mon[66455]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-mon[66455]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:59:12.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:12 smithi032 ceph-mon[66455]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T11:59:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:13 smithi146 ceph-mon[53008]: pgmap v298: 97 pgs: 1 remapped+peering, 4 peering, 6 active+undersized, 22 active+undersized+degraded, 64 active+clean; 245 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 110 KiB/s wr, 37 op/s; 64/507 objects degraded (12.623%); 0 B/s, 7 objects/s recovering 2024-03-24T11:59:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:13 smithi146 ceph-mon[53008]: Health check update: Degraded data redundancy: 64/507 objects degraded (12.623%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:13 smithi146 ceph-mon[53008]: fsmap foofs:1/1 {0=foofs.smithi146.xsrsue=up:replay} 2024-03-24T11:59:13.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:13 smithi146 ceph-mon[53008]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:59:13.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:13 smithi032 ceph-mon[66455]: pgmap v298: 97 pgs: 1 remapped+peering, 4 peering, 6 active+undersized, 22 active+undersized+degraded, 64 active+clean; 245 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 110 KiB/s wr, 37 op/s; 64/507 objects degraded (12.623%); 0 B/s, 7 objects/s recovering 2024-03-24T11:59:13.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:13 smithi032 ceph-mon[66455]: Health check update: Degraded data redundancy: 64/507 objects degraded (12.623%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T11:59:13.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:13 smithi032 ceph-mon[66455]: fsmap foofs:1/1 {0=foofs.smithi146.xsrsue=up:replay} 2024-03-24T11:59:13.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:13 smithi032 ceph-mon[66455]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:59:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:15 smithi146 ceph-mon[53008]: pgmap v301: 97 pgs: 5 peering, 1 active+undersized+degraded, 91 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 15 KiB/s wr, 16 op/s; 6/432 objects degraded (1.389%); 2.5 MiB/s, 79 objects/s recovering 2024-03-24T11:59:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:15 smithi146 ceph-mon[53008]: reconnect by client.24343 172.21.15.146:0/2998143345 after 0 2024-03-24T11:59:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:15 smithi146 ceph-mon[53008]: reconnect by client.14552 172.21.15.32:0/2572480292 after 0 2024-03-24T11:59:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:15 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:reconnect 2024-03-24T11:59:15.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:15 smithi146 ceph-mon[53008]: fsmap foofs:1/1 {0=foofs.smithi146.xsrsue=up:reconnect} 2024-03-24T11:59:15.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:15 smithi032 ceph-mon[66455]: pgmap v301: 97 pgs: 5 peering, 1 active+undersized+degraded, 91 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 15 KiB/s wr, 16 op/s; 6/432 objects degraded (1.389%); 2.5 MiB/s, 79 objects/s recovering 2024-03-24T11:59:15.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:15 smithi032 ceph-mon[66455]: reconnect by client.24343 172.21.15.146:0/2998143345 after 0 2024-03-24T11:59:15.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:15 smithi032 ceph-mon[66455]: reconnect by client.14552 172.21.15.32:0/2572480292 after 0 2024-03-24T11:59:15.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:15 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:reconnect 2024-03-24T11:59:15.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:15 smithi032 ceph-mon[66455]: fsmap foofs:1/1 {0=foofs.smithi146.xsrsue=up:reconnect} 2024-03-24T11:59:16.612 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:16 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:rejoin 2024-03-24T11:59:16.612 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:16 smithi146 ceph-mon[53008]: fsmap foofs:1/1 {0=foofs.smithi146.xsrsue=up:rejoin} 2024-03-24T11:59:16.612 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:16 smithi146 ceph-mon[53008]: daemon mds.foofs.smithi146.xsrsue is now active in filesystem foofs as rank 0 2024-03-24T11:59:16.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:16 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:rejoin 2024-03-24T11:59:16.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:16 smithi032 ceph-mon[66455]: fsmap foofs:1/1 {0=foofs.smithi146.xsrsue=up:rejoin} 2024-03-24T11:59:16.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:16 smithi032 ceph-mon[66455]: daemon mds.foofs.smithi146.xsrsue is now active in filesystem foofs as rank 0 2024-03-24T11:59:17.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:17 smithi146 ceph-mon[53008]: pgmap v302: 97 pgs: 1 peering, 96 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 11 KiB/s wr, 14 op/s; 1.8 MiB/s, 67 objects/s recovering 2024-03-24T11:59:17.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:17 smithi146 ceph-mon[53008]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/432 objects degraded (1.389%), 1 pg degraded) 2024-03-24T11:59:17.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:17 smithi146 ceph-mon[53008]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T11:59:17.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:17 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:active 2024-03-24T11:59:17.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:17 smithi146 ceph-mon[53008]: fsmap foofs:1 {0=foofs.smithi146.xsrsue=up:active} 2024-03-24T11:59:17.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:17 smithi032 ceph-mon[66455]: pgmap v302: 97 pgs: 1 peering, 96 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 11 KiB/s wr, 14 op/s; 1.8 MiB/s, 67 objects/s recovering 2024-03-24T11:59:17.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:17 smithi032 ceph-mon[66455]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/432 objects degraded (1.389%), 1 pg degraded) 2024-03-24T11:59:17.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:17 smithi032 ceph-mon[66455]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T11:59:17.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:17 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.146:6832/1246747794,v1:172.21.15.146:6833/1246747794] up:active 2024-03-24T11:59:17.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:17 smithi032 ceph-mon[66455]: fsmap foofs:1 {0=foofs.smithi146.xsrsue=up:active} 2024-03-24T11:59:19.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:19 smithi146 ceph-mon[53008]: pgmap v303: 97 pgs: 1 peering, 96 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 9.3 KiB/s wr, 14 op/s; 1.5 MiB/s, 52 objects/s recovering 2024-03-24T11:59:19.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:19 smithi032 ceph-mon[66455]: pgmap v303: 97 pgs: 1 peering, 96 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 9.3 KiB/s wr, 14 op/s; 1.5 MiB/s, 52 objects/s recovering 2024-03-24T11:59:21.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:21 smithi146 ceph-mon[53008]: pgmap v304: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 7.7 KiB/s wr, 18 op/s; 1.2 MiB/s, 42 objects/s recovering 2024-03-24T11:59:21.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:21 smithi032 ceph-mon[66455]: pgmap v304: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 7.7 KiB/s wr, 18 op/s; 1.2 MiB/s, 42 objects/s recovering 2024-03-24T11:59:23.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:23 smithi032 ceph-mon[66455]: pgmap v305: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 6.6 KiB/s wr, 16 op/s; 1.0 MiB/s, 35 objects/s recovering 2024-03-24T11:59:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:23 smithi146 ceph-mon[53008]: pgmap v305: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 6.6 KiB/s wr, 16 op/s; 1.0 MiB/s, 35 objects/s recovering 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: pgmap v306: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 7.3 KiB/s wr, 14 op/s; 392 KiB/s, 26 objects/s recovering 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:boot 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: fsmap foofs:1 {0=foofs.smithi146.xsrsue=up:active} 1 up:standby 2024-03-24T11:59:25.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: pgmap v306: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 7.3 KiB/s wr, 14 op/s; 392 KiB/s, 26 objects/s recovering 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:boot 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: fsmap foofs:1 {0=foofs.smithi146.xsrsue=up:active} 1 up:standby 2024-03-24T11:59:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:59:26.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:26.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:26.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:26 smithi032 ceph-mon[66455]: pgmap v307: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.3 KiB/s wr, 11 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:59:26.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:26 smithi146 ceph-mon[53008]: pgmap v307: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.3 KiB/s wr, 11 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:59:27.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:28.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:28 smithi032 ceph-mon[66455]: pgmap v308: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-24T11:59:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:28 smithi146 ceph-mon[53008]: pgmap v308: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-24T11:59:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:30 smithi146 ceph-mon[53008]: pgmap v309: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 7.6 KiB/s wr, 8 op/s 2024-03-24T11:59:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:30 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:31.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:30 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:30 smithi032 ceph-mon[66455]: pgmap v309: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 7.6 KiB/s wr, 8 op/s 2024-03-24T11:59:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:30 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:30 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:32.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:31 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:32.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:31 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:32.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:31 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:32.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:31 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:33.334 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:33 smithi032 ceph-mon[66455]: pgmap v310: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 7.4 KiB/s wr, 3 op/s 2024-03-24T11:59:33.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:33 smithi146 ceph-mon[53008]: pgmap v310: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 7.4 KiB/s wr, 3 op/s 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: pgmap v311: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.6 KiB/s wr, 3 op/s 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:35.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi146.xsrsue"]}]: dispatch 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: pgmap v311: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.6 KiB/s wr, 3 op/s 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:35.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi146.xsrsue"]}]: dispatch 2024-03-24T11:59:36.271 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:36 smithi146 ceph-mon[53008]: Detected new or changed devices on smithi032 2024-03-24T11:59:36.272 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:36 smithi146 ceph-mon[53008]: Upgrade: It appears safe to stop mds.foofs.smithi146.xsrsue 2024-03-24T11:59:36.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:36 smithi032 ceph-mon[66455]: Detected new or changed devices on smithi032 2024-03-24T11:59:36.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:36 smithi032 ceph-mon[66455]: Upgrade: It appears safe to stop mds.foofs.smithi146.xsrsue 2024-03-24T11:59:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:37 smithi146 ceph-mon[53008]: pgmap v312: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 1.4 KiB/s wr, 0 op/s 2024-03-24T11:59:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:37 smithi146 ceph-mon[53008]: Upgrade: Updating mds.foofs.smithi146.xsrsue 2024-03-24T11:59:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:59:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:37.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:37 smithi146 ceph-mon[53008]: Deploying daemon mds.foofs.smithi146.xsrsue on smithi146 2024-03-24T11:59:37.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:37 smithi032 ceph-mon[66455]: pgmap v312: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 1.4 KiB/s wr, 0 op/s 2024-03-24T11:59:37.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:37 smithi032 ceph-mon[66455]: Upgrade: Updating mds.foofs.smithi146.xsrsue 2024-03-24T11:59:37.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:37.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi146.xsrsue", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:59:37.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:37.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:37 smithi032 ceph-mon[66455]: Deploying daemon mds.foofs.smithi146.xsrsue on smithi146 2024-03-24T11:59:39.134 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 11:59:38 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T11:59:38.730+0000 7f63cd1c1700 -1 bdev(0x564372f10000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 5 2024-03-24T11:59:39.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:38 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[66451]: 2024-03-24T11:59:38.708+0000 7efe750aa700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: pgmap v313: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 1.4 KiB/s wr, 0 op/s 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: Standby daemon mds.foofs.smithi032.qzdcjy assigned to filesystem foofs as rank 0 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:59:39.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T11:59:39.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:39 smithi146 ceph-mon[53008]: fsmap foofs:1/1 {0=foofs.smithi032.qzdcjy=up:replay} 2024-03-24T11:59:39.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: pgmap v313: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 1.4 KiB/s wr, 0 op/s 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: Standby daemon mds.foofs.smithi032.qzdcjy assigned to filesystem foofs as rank 0 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T11:59:39.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:39 smithi032 ceph-mon[66455]: fsmap foofs:1/1 {0=foofs.smithi032.qzdcjy=up:replay} 2024-03-24T11:59:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:40 smithi146 ceph-mon[53008]: reconnect by client.14552 172.21.15.32:0/2572480292 after 0 2024-03-24T11:59:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:40 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:reconnect 2024-03-24T11:59:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:40 smithi146 ceph-mon[53008]: fsmap foofs:1/1 {0=foofs.smithi032.qzdcjy=up:reconnect} 2024-03-24T11:59:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:40 smithi146 ceph-mon[53008]: pgmap v315: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 204 B/s wr, 1 op/s 2024-03-24T11:59:41.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:40 smithi146 ceph-mon[53008]: reconnect by client.24343 172.21.15.146:0/2998143345 after 0.0529991 2024-03-24T11:59:41.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:40 smithi032 ceph-mon[66455]: reconnect by client.14552 172.21.15.32:0/2572480292 after 0 2024-03-24T11:59:41.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:40 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:reconnect 2024-03-24T11:59:41.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:40 smithi032 ceph-mon[66455]: fsmap foofs:1/1 {0=foofs.smithi032.qzdcjy=up:reconnect} 2024-03-24T11:59:41.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:40 smithi032 ceph-mon[66455]: pgmap v315: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 204 B/s wr, 1 op/s 2024-03-24T11:59:41.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:40 smithi032 ceph-mon[66455]: reconnect by client.24343 172.21.15.146:0/2998143345 after 0.0529991 2024-03-24T11:59:41.434 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T11:59:42.107 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (10m) 12s ago 19m 22.6M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (6m) 12s ago 19m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (6m) 40s ago 18m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (9m) 12s ago 18m 53.1M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (18s) 12s ago 13m 14.6M - 19.0.0-1667-gdb0330b1 4f325f8440e9 95c8d849ddcd 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (13m) 40s ago 13m 17.5M - 17.2.0 e1d6a67b021e 464ddb93a07e 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (9m) 12s ago 20m 511M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (8m) 40s ago 18m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 12s ago 20m 59.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (7m) 40s ago 18m 43.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (13m) 12s ago 13m 256M - 4.0 e1d6a67b021e 1f7968537297 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (12m) 40s ago 12m 91.1M - 4.0 e1d6a67b021e a51bc19cb004 2024-03-24T11:59:42.108 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (9m) 12s ago 18m 15.4M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (9m) 40s ago 18m 17.6M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (5m) 12s ago 17m 429M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 12s ago 17m 394M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 12s ago 16m 353M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (3m) 12s ago 16m 437M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (2m) 40s ago 16m 340M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (2m) 40s ago 15m 297M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (84s) 40s ago 15m 276M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (46s) 40s ago 14m 14.8M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 85b15c0b6114 2024-03-24T11:59:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8m) 12s ago 18m 69.0M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T11:59:42.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:41 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:rejoin 2024-03-24T11:59:42.109 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:41 smithi032 ceph-mon[66455]: fsmap foofs:1/1 {0=foofs.smithi032.qzdcjy=up:rejoin} 2024-03-24T11:59:42.110 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:42.110 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:42.110 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:41 smithi032 ceph-mon[66455]: daemon mds.foofs.smithi032.qzdcjy is now active in filesystem foofs as rank 0 2024-03-24T11:59:42.110 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:41 smithi032 ceph-mon[66455]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:42.110 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:59:42.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:41 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:rejoin 2024-03-24T11:59:42.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:41 smithi146 ceph-mon[53008]: fsmap foofs:1/1 {0=foofs.smithi032.qzdcjy=up:rejoin} 2024-03-24T11:59:42.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:42.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:42.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:41 smithi146 ceph-mon[53008]: daemon mds.foofs.smithi032.qzdcjy is now active in filesystem foofs as rank 0 2024-03-24T11:59:42.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:41 smithi146 ceph-mon[53008]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:42.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T11:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 13 2024-03-24T11:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T11:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:59:42.857 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T11:59:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/23 daemons upgraded", 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T11:59:42.858 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T11:59:42.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:42 smithi032 ceph-mon[66455]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T11:59:42.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:42 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:active 2024-03-24T11:59:42.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:42 smithi032 ceph-mon[66455]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 2024-03-24T11:59:42.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:42 smithi032 ceph-mon[66455]: pgmap v316: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 204 B/s wr, 3 op/s 2024-03-24T11:59:42.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:42 smithi032 ceph-mon[66455]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:42.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:42 smithi032 ceph-mon[66455]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:42.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:42 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3199586917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:43.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:42 smithi146 ceph-mon[53008]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T11:59:43.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:42 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485] up:active 2024-03-24T11:59:43.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:42 smithi146 ceph-mon[53008]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 2024-03-24T11:59:43.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:42 smithi146 ceph-mon[53008]: pgmap v316: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 204 B/s wr, 3 op/s 2024-03-24T11:59:43.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:42 smithi146 ceph-mon[53008]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:43.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:42 smithi146 ceph-mon[53008]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:43.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:42 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3199586917' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:43.264 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN insufficient standby MDS daemons available 2024-03-24T11:59:43.264 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available 2024-03-24T11:59:43.264 INFO:teuthology.orchestra.run.smithi032.stdout: have 0; want 1 more 2024-03-24T11:59:44.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:43 smithi146 ceph-mon[53008]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:44.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:43 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/2425526271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:59:44.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:43 smithi032 ceph-mon[66455]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:59:44.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:43 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2425526271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:59:45.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:44 smithi146 ceph-mon[53008]: pgmap v317: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 204 B/s wr, 8 op/s 2024-03-24T11:59:45.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:44 smithi032 ceph-mon[66455]: pgmap v317: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 204 B/s wr, 8 op/s 2024-03-24T11:59:47.091 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:46 smithi146 ceph-mon[53008]: pgmap v318: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T11:59:47.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:46 smithi032 ceph-mon[66455]: pgmap v318: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T11:59:49.181 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: pgmap v319: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T11:59:49.181 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:49.181 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:49.181 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:49.181 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T11:59:49.182 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: Cluster is now healthy 2024-03-24T11:59:49.182 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: mds.? [v2:172.21.15.146:6832/3000201445,v1:172.21.15.146:6833/3000201445] up:boot 2024-03-24T11:59:49.182 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:59:49.182 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:49 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: pgmap v319: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: Cluster is now healthy 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: mds.? [v2:172.21.15.146:6832/3000201445,v1:172.21.15.146:6833/3000201445] up:boot 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: fsmap foofs:1 {0=foofs.smithi032.qzdcjy=up:active} 1 up:standby 2024-03-24T11:59:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:49 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:59:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:51 smithi146 ceph-mon[53008]: pgmap v320: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 7.0 KiB/s wr, 13 op/s 2024-03-24T11:59:51.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:51 smithi032 ceph-mon[66455]: pgmap v320: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 7.0 KiB/s wr, 13 op/s 2024-03-24T11:59:52.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:52 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:52.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:52 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:52.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:52 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:52.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:52 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:53.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:53 smithi146 ceph-mon[53008]: pgmap v321: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.4 KiB/s wr, 11 op/s 2024-03-24T11:59:53.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:53 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:53.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:53 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:53.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:53 smithi032 ceph-mon[66455]: pgmap v321: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 6.4 KiB/s wr, 11 op/s 2024-03-24T11:59:53.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:53 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:53.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:53 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:55.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:55 smithi146 ceph-mon[53008]: pgmap v322: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-03-24T11:59:55.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:55 smithi032 ceph-mon[66455]: pgmap v322: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-03-24T11:59:56.741 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: Detected new or changed devices on smithi146 2024-03-24T11:59:56.741 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.741 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.741 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:56.741 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.741 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: pgmap v323: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all mds 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:59:56.742 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi032.qzdcjy"}]': finished 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi146.xsrsue"}]': finished 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all rgw 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all rbd-mirror 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all ceph-exporter 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.743 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:56.744 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all iscsi 2024-03-24T11:59:56.744 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:56.744 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-24T11:59:56.744 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.4"}]: dispatch 2024-03-24T11:59:56.744 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-03-24T11:59:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: Detected new or changed devices on smithi146 2024-03-24T11:59:57.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: pgmap v323: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all mds 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi032.qzdcjy"}]: dispatch 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi032.qzdcjy"}]': finished 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi146.xsrsue"}]: dispatch 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi146.xsrsue"}]': finished 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all rgw 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all rbd-mirror 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all ceph-exporter 2024-03-24T11:59:57.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:59:57.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all iscsi 2024-03-24T11:59:57.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:57.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-24T11:59:57.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.4"}]: dispatch 2024-03-24T11:59:57.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.4"}]': finished 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]': finished 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: Upgrade: Updating nfs.foo.0.0.smithi032.lizmvs (1/2) 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs 2024-03-24T11:59:58.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T11:59:58.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T11:59:58.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T11:59:58.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T11:59:58.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:58.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-24T11:59:58.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.4"}]': finished 2024-03-24T11:59:58.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]': finished 2024-03-24T11:59:58.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:59:58.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: Upgrade: Updating nfs.foo.0.0.smithi032.lizmvs (1/2) 2024-03-24T11:59:58.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T11:59:58.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs 2024-03-24T11:59:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T11:59:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T11:59:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T11:59:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T11:59:58.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:59.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: pgmap v325: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 9.0 KiB/s wr, 3 op/s 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: Rados config object exists: conf-nfs.foo 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs-rgw 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: Bind address in nfs.foo.0.0.smithi032.lizmvs's ganesha conf is defaulting to empty 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:59.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 11:59:58 smithi146 ceph-mon[53008]: Deploying daemon nfs.foo.0.0.smithi032.lizmvs on smithi032 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: pgmap v325: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 9.0 KiB/s wr, 3 op/s 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: Rados config object exists: conf-nfs.foo 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: Creating key for client.nfs.foo.0.0.smithi032.lizmvs-rgw 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi032.lizmvs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: Bind address in nfs.foo.0.0.smithi032.lizmvs's ganesha conf is defaulting to empty 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:59.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 11:59:58 smithi032 ceph-mon[66455]: Deploying daemon nfs.foo.0.0.smithi032.lizmvs on smithi032 2024-03-24T12:00:01.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:00 smithi032 ceph-mon[66455]: pgmap v327: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s wr, 0 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:01.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:00 smithi032 ceph-mon[66455]: overall HEALTH_OK 2024-03-24T12:00:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:00 smithi146 ceph-mon[53008]: pgmap v327: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s wr, 0 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:00 smithi146 ceph-mon[53008]: overall HEALTH_OK 2024-03-24T12:00:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:02 smithi146 ceph-mon[53008]: pgmap v328: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:02 smithi146 ceph-mon[53008]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.32:0/2572480292"}]: dispatch 2024-03-24T12:00:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:02 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:03.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:02 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:03.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:02 smithi032 ceph-mon[66455]: pgmap v328: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:03.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:02 smithi032 ceph-mon[66455]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.32:0/2572480292"}]: dispatch 2024-03-24T12:00:03.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:03.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:02 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: Evicting (and blocklisting) client session 14552 (172.21.15.32:0/2572480292) 2024-03-24T12:00:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.32:0/2572480292"}]': finished 2024-03-24T12:00:04.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T12:00:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T12:00:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T12:00:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T12:00:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:04.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T12:00:04.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:03 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T12:00:04.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: Evicting (and blocklisting) client session 14552 (172.21.15.32:0/2572480292) 2024-03-24T12:00:04.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.32:0/2572480292"}]': finished 2024-03-24T12:00:04.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T12:00:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T12:00:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T12:00:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T12:00:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T12:00:04.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:03 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T12:00:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: pgmap v330: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: Upgrade: Updating nfs.foo.1.0.smithi146.rjcwds (2/2) 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: Rados config object exists: conf-nfs.foo 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds-rgw 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: Bind address in nfs.foo.1.0.smithi146.rjcwds's ganesha conf is defaulting to empty 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:05.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:04 smithi146 ceph-mon[53008]: Deploying daemon nfs.foo.1.0.smithi146.rjcwds on smithi146 2024-03-24T12:00:05.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: pgmap v330: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: Upgrade: Updating nfs.foo.1.0.smithi146.rjcwds (2/2) 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: Rados config object exists: conf-nfs.foo 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: Creating key for client.nfs.foo.1.0.smithi146.rjcwds-rgw 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi146.rjcwds-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: Bind address in nfs.foo.1.0.smithi146.rjcwds's ganesha conf is defaulting to empty 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:05.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:04 smithi032 ceph-mon[66455]: Deploying daemon nfs.foo.1.0.smithi146.rjcwds on smithi146 2024-03-24T12:00:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:06 smithi146 ceph-mon[53008]: pgmap v331: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s wr, 1 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:07.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:06 smithi032 ceph-mon[66455]: pgmap v331: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s wr, 1 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:09.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:08 smithi146 ceph-mon[53008]: pgmap v332: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 935 B/s wr, 1 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:09.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:09.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:09.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:09.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:08 smithi032 ceph-mon[66455]: pgmap v332: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 935 B/s wr, 1 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T12:00:09.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:09.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:09.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:10 smithi146 ceph-mon[53008]: pgmap v333: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T12:00:11.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:10 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:11.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:10 smithi032 ceph-mon[66455]: pgmap v333: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T12:00:11.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:10 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:12.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:11 smithi032 ceph-mon[66455]: Evicting (and blocklisting) client session 24343 (172.21.15.146:0/2998143345) 2024-03-24T12:00:12.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:11 smithi032 ceph-mon[66455]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.146:0/2998143345"}]: dispatch 2024-03-24T12:00:12.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:12.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:11 smithi146 ceph-mon[53008]: Evicting (and blocklisting) client session 24343 (172.21.15.146:0/2998143345) 2024-03-24T12:00:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:11 smithi146 ceph-mon[53008]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.146:0/2998143345"}]: dispatch 2024-03-24T12:00:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:12.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:12 smithi032 ceph-mon[66455]: pgmap v334: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T12:00:12.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:12 smithi032 ceph-mon[66455]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.146:0/2998143345"}]': finished 2024-03-24T12:00:12.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:12 smithi032 ceph-mon[66455]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T12:00:12.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:12 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:12.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:12 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:12.983 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:12 smithi146 ceph-mon[53008]: pgmap v334: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T12:00:12.983 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:12 smithi146 ceph-mon[53008]: from='mds.? [v2:172.21.15.32:6834/2654930485,v1:172.21.15.32:6835/2654930485]' entity='mds.foofs.smithi032.qzdcjy' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.146:0/2998143345"}]': finished 2024-03-24T12:00:12.983 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:12 smithi146 ceph-mon[53008]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T12:00:12.983 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:12 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:12.983 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:12 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:13.641 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T12:00:14.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:14 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:14.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:14 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:14.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:14 smithi146 ceph-mon[53008]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:14.390 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T12:00:14.390 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (10m) 1s ago 19m 22.7M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T12:00:14.390 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 1s ago 19m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T12:00:14.390 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (6m) 3s ago 19m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T12:00:14.390 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (10m) 1s ago 19m 54.2M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T12:00:14.390 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (50s) 1s ago 13m 35.7M - 19.0.0-1667-gdb0330b1 4f325f8440e9 95c8d849ddcd 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (26s) 3s ago 13m 11.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 976a8ba8b4ad 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (9m) 1s ago 21m 512M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (8m) 3s ago 19m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 1s ago 21m 61.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (7m) 3s ago 18m 46.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (12s) 1s ago 13m 61.3M - 5.7 4f325f8440e9 c8d78602083c 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (7s) 3s ago 13m 15.0M - 5.7 4f325f8440e9 21450c0428e3 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (10m) 1s ago 19m 20.2M - 1.3.1 1dbe0e931976 9f9550c26bce 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (9m) 3s ago 18m 17.9M - 1.3.1 1dbe0e931976 e8d66c779d4d 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 1s ago 18m 430M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T12:00:14.391 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 1s ago 17m 396M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T12:00:14.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 1s ago 17m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T12:00:14.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (3m) 1s ago 16m 438M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T12:00:14.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (3m) 3s ago 16m 342M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T12:00:14.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (2m) 3s ago 16m 307M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T12:00:14.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (116s) 3s ago 15m 290M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T12:00:14.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (79s) 3s ago 15m 301M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 85b15c0b6114 2024-03-24T12:00:14.392 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (9m) 1s ago 18m 71.1M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T12:00:14.392 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:14 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:14.392 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:14 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:14.393 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:14 smithi032 ceph-mon[66455]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:14.805 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:00:14.805 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T12:00:14.805 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:00:14.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:14.805 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T12:00:14.805 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:00:14.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T12:00:14.806 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:00:15.167 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "nfs", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "18/23 daemons upgraded", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading nfs daemons", 2024-03-24T12:00:15.168 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T12:00:15.169 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:00:15.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:15 smithi032 ceph-mon[66455]: pgmap v336: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T12:00:15.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:15 smithi032 ceph-mon[66455]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:15.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:15 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:15.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:15 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:15.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:15 smithi032 ceph-mon[66455]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:15.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:15 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/1009844035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:15.544 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:15 smithi146 ceph-mon[53008]: pgmap v336: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T12:00:15.544 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:15 smithi146 ceph-mon[53008]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:15.544 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:15 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:15.544 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:15 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:15.544 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:15 smithi146 ceph-mon[53008]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:15.544 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:15 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/1009844035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:15.581 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T12:00:16.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:16 smithi032 ceph-mon[66455]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:16.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:16 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2202956629' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:00:16.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:16.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:16 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:16.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:16 smithi146 ceph-mon[53008]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:16.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:16 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/2202956629' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:00:16.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:16.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:16 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:17.816 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:17 smithi032 ceph-mon[66455]: pgmap v337: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T12:00:17.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:17 smithi146 ceph-mon[53008]: pgmap v337: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T12:00:19.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: pgmap v338: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T12:00:19.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all nfs 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi032.lizmvs"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi032.lizmvs"}]': finished 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi146.rjcwds"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi146.rjcwds"}]': finished 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: Upgrade: Setting container_image for all nvmeof 2024-03-24T12:00:19.316 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:19 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: pgmap v338: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all nfs 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi032.lizmvs"}]: dispatch 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi032.lizmvs"}]': finished 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi146.rjcwds"}]: dispatch 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi146.rjcwds"}]': finished 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: Upgrade: Setting container_image for all nvmeof 2024-03-24T12:00:19.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:19 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:21.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:21 smithi032 ceph-mon[66455]: pgmap v339: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T12:00:21.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:21 smithi032 ceph-mon[66455]: Upgrade: Updating node-exporter.smithi032 (1/2) 2024-03-24T12:00:21.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:21 smithi032 ceph-mon[66455]: Deploying daemon node-exporter.smithi032 on smithi032 2024-03-24T12:00:21.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:21 smithi146 ceph-mon[53008]: pgmap v339: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T12:00:21.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:21 smithi146 ceph-mon[53008]: Upgrade: Updating node-exporter.smithi032 (1/2) 2024-03-24T12:00:21.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:21 smithi146 ceph-mon[53008]: Deploying daemon node-exporter.smithi032 on smithi032 2024-03-24T12:00:23.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:23 smithi032 ceph-mon[66455]: pgmap v340: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T12:00:23.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:23 smithi146 ceph-mon[53008]: pgmap v340: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T12:00:25.562 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:25 smithi032 ceph-mon[66455]: pgmap v341: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 345 B/s rd, 345 B/s wr, 0 op/s 2024-03-24T12:00:25.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:25 smithi146 ceph-mon[53008]: pgmap v341: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 345 B/s rd, 345 B/s wr, 0 op/s 2024-03-24T12:00:26.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:26.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:26.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:26.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:26.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:26.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:27.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:27 smithi146 ceph-mon[53008]: pgmap v342: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:27.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:27 smithi032 ceph-mon[66455]: pgmap v342: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:28.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:28 smithi146 ceph-mon[53008]: Upgrade: Updating node-exporter.smithi146 (2/2) 2024-03-24T12:00:28.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:28 smithi146 ceph-mon[53008]: Deploying daemon node-exporter.smithi146 on smithi146 2024-03-24T12:00:28.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:28 smithi032 ceph-mon[66455]: Upgrade: Updating node-exporter.smithi146 (2/2) 2024-03-24T12:00:28.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:28 smithi032 ceph-mon[66455]: Deploying daemon node-exporter.smithi146 on smithi146 2024-03-24T12:00:29.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:29 smithi146 ceph-mon[53008]: pgmap v343: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:29.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:29 smithi032 ceph-mon[66455]: pgmap v343: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:31.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:31 smithi146 ceph-mon[53008]: pgmap v344: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:31.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:31 smithi032 ceph-mon[66455]: pgmap v344: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:33 smithi146 ceph-mon[53008]: pgmap v345: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:33.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:33 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:33.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:33 smithi032 ceph-mon[66455]: pgmap v345: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:33.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:33.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:33.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:33.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:33.704 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:33 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:35 smithi146 ceph-mon[53008]: pgmap v346: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:00:35.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:35 smithi032 ceph-mon[66455]: pgmap v346: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:00:36.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:36 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:36.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:36 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:36.690 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:36 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:36.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:36 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:37 smithi146 ceph-mon[53008]: pgmap v347: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:37.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:37.679 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:37 smithi032 ceph-mon[66455]: pgmap v347: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:37.679 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:37.679 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:39.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:38 smithi146 ceph-mon[53008]: pgmap v348: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:39.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:38 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:39.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:38 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:39.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:38 smithi032 ceph-mon[66455]: pgmap v348: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:39.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:38 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:39.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:38 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:40 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:40.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:40 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:41.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:41 smithi032 ceph-mon[66455]: pgmap v349: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:41.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:41.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:41 smithi146 ceph-mon[53008]: pgmap v349: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:41.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: pgmap v350: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.692 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:43 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: pgmap v350: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:43.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.885 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:43.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:43 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:45.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:45 smithi032 ceph-mon[66455]: pgmap v351: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:00:45.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:45 smithi032 ceph-mon[66455]: Upgrade: Updating prometheus.smithi032 2024-03-24T12:00:45.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:45 smithi032 ceph-mon[66455]: Deploying daemon prometheus.smithi032 on smithi032 2024-03-24T12:00:45.939 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T12:00:46.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:45 smithi146 ceph-mon[53008]: pgmap v351: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:00:46.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:45 smithi146 ceph-mon[53008]: Upgrade: Updating prometheus.smithi032 2024-03-24T12:00:46.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:45 smithi146 ceph-mon[53008]: Deploying daemon prometheus.smithi032 on smithi032 2024-03-24T12:00:46.618 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T12:00:46.618 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (11m) 8s ago 20m 22.7M - 0.23.0 ba2b418f427c a2c0d0e7b613 2024-03-24T12:00:46.618 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 8s ago 20m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (7m) 11s ago 19m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (11m) 8s ago 20m 54.6M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (82s) 8s ago 14m 36.0M - 19.0.0-1667-gdb0330b1 4f325f8440e9 95c8d849ddcd 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (59s) 11s ago 14m 11.5M - 19.0.0-1667-gdb0330b1 4f325f8440e9 976a8ba8b4ad 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (10m) 8s ago 21m 513M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (9m) 11s ago 19m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 8s ago 21m 64.8M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (8m) 11s ago 19m 50.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (44s) 8s ago 14m 61.8M - 5.7 4f325f8440e9 c8d78602083c 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (39s) 11s ago 13m 57.2M - 5.7 4f325f8440e9 21450c0428e3 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (21s) 8s ago 19m 5301k - 1.5.0 0da6a335fe13 f5d342c5eb41 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (14s) 11s ago 19m 3313k - 1.5.0 0da6a335fe13 63e203152582 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 8s ago 18m 431M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T12:00:46.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 8s ago 18m 396M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T12:00:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 8s ago 17m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T12:00:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (4m) 8s ago 17m 439M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T12:00:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (3m) 11s ago 17m 342M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T12:00:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (3m) 11s ago 16m 307M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T12:00:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (2m) 11s ago 16m 291M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T12:00:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (111s) 11s ago 15m 301M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 85b15c0b6114 2024-03-24T12:00:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (9m) 8s ago 19m 71.1M - 2.33.4 514e6a882f6e 1eaa1e8835df 2024-03-24T12:00:47.026 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:00:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T12:00:47.028 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T12:00:47.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T12:00:47.028 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T12:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T12:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T12:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T12:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "nfs", 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "18/23 daemons upgraded", 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T12:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:00:47.770 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T12:00:47.936 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:47 smithi146 ceph-mon[53008]: pgmap v352: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:47.936 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:47 smithi146 ceph-mon[53008]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:47.936 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:47 smithi146 ceph-mon[53008]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:47.936 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:47 smithi146 ceph-mon[53008]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:47.936 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:47 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/1492342061' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:47.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:47 smithi032 ceph-mon[66455]: pgmap v352: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:47.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:47 smithi032 ceph-mon[66455]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:47.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:47 smithi032 ceph-mon[66455]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:47.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:47 smithi032 ceph-mon[66455]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:47.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:47 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/1492342061' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:00:48.908 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:48 smithi032 ceph-mon[66455]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:48.909 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:48 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2161365854' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:00:49.111 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:48 smithi146 ceph-mon[53008]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:00:49.111 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:48 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/2161365854' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:00:49.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:49 smithi032 ceph-mon[66455]: pgmap v353: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:50.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:49 smithi146 ceph-mon[53008]: pgmap v353: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:50.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:50 smithi032 ceph-mon[66455]: pgmap v354: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:50.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:50.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:50.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:50 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:50 smithi146 ceph-mon[53008]: pgmap v354: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:50 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:53.059 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:52 smithi032 ceph-mon[66455]: pgmap v355: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:52 smithi146 ceph-mon[53008]: pgmap v355: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:55.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:54 smithi032 ceph-mon[66455]: pgmap v356: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:00:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:54 smithi146 ceph-mon[53008]: pgmap v356: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:00:56.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:56.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:56.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:55 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:56.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:56.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:56.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:55 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:57.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:57 smithi146 ceph-mon[53008]: pgmap v357: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:57.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:57.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:57 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:57.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:57 smithi032 ceph-mon[66455]: pgmap v357: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:57.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:57.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:57 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:00:59.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:00:59 smithi146 ceph-mon[53008]: pgmap v358: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:00:59.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:00:59 smithi032 ceph-mon[66455]: pgmap v358: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: pgmap v359: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:00.886 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.887 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:00.888 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:00 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.130 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: pgmap v359: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.131 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:01.132 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:00 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:02.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:02 smithi032 ceph-mon[66455]: Upgrade: Updating alertmanager.smithi032 2024-03-24T12:01:02.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:02 smithi032 ceph-mon[66455]: pgmap v360: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:02.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:02 smithi032 ceph-mon[66455]: Deploying daemon alertmanager.smithi032 on smithi032 2024-03-24T12:01:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:02 smithi146 ceph-mon[53008]: Upgrade: Updating alertmanager.smithi032 2024-03-24T12:01:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:02 smithi146 ceph-mon[53008]: pgmap v360: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:03.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:02 smithi146 ceph-mon[53008]: Deploying daemon alertmanager.smithi032 on smithi032 2024-03-24T12:01:05.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:04 smithi146 ceph-mon[53008]: pgmap v361: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:05.182 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:04 smithi032 ceph-mon[66455]: pgmap v361: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:06.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:06 smithi032 ceph-mon[66455]: pgmap v362: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:07.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:06 smithi146 ceph-mon[53008]: pgmap v362: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:08.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:08.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:08.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:08 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:08.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:08.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:08.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:08 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:09.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:09 smithi146 ceph-mon[53008]: pgmap v363: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:09.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:09 smithi032 ceph-mon[66455]: pgmap v363: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:11.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:11 smithi032 ceph-mon[66455]: pgmap v364: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:11.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:11 smithi146 ceph-mon[53008]: pgmap v364: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:13.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:13 smithi032 ceph-mon[66455]: pgmap v365: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:13.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:13 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:13.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:13 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:13 smithi146 ceph-mon[53008]: pgmap v365: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:13 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:13.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:13 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:15.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:14 smithi032 ceph-mon[66455]: pgmap v366: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:15.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:14 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:15.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:14 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:14 smithi146 ceph-mon[53008]: pgmap v366: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:14 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:14 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:17.121 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:16 smithi032 ceph-mon[66455]: pgmap v367: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:17.186 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:16 smithi146 ceph-mon[53008]: pgmap v367: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:18.144 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T12:01:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.635 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.636 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:18 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.703 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:18 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:18.829 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T12:01:18.829 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (12s) 6s ago 20m 15.9M - 0.25.0 c8568f914cd2 7e89a3c078b6 2024-03-24T12:01:18.829 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 6s ago 20m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T12:01:18.829 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (8m) 43s ago 20m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (11m) 6s ago 20m 54.5M - 8.3.5 dad864ee21e9 d13e3693b3ae 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (115s) 6s ago 14m 36.2M - 19.0.0-1667-gdb0330b1 4f325f8440e9 95c8d849ddcd 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (91s) 43s ago 14m 11.5M - 19.0.0-1667-gdb0330b1 4f325f8440e9 976a8ba8b4ad 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (10m) 6s ago 22m 516M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (9m) 43s ago 20m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 6s ago 22m 62.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (9m) 43s ago 20m 50.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (77s) 6s ago 14m 62.3M - 5.7 4f325f8440e9 c8d78602083c 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (71s) 43s ago 14m 57.2M - 5.7 4f325f8440e9 21450c0428e3 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (53s) 6s ago 20m 8170k - 1.5.0 0da6a335fe13 f5d342c5eb41 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (47s) 43s ago 19m 3313k - 1.5.0 0da6a335fe13 63e203152582 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 6s ago 19m 431M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T12:01:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 6s ago 18m 396M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T12:01:18.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 6s ago 18m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T12:01:18.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (5m) 6s ago 18m 438M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T12:01:18.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (4m) 43s ago 17m 342M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T12:01:18.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (3m) 43s ago 17m 307M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T12:01:18.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (3m) 43s ago 16m 291M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T12:01:18.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (2m) 43s ago 16m 301M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 85b15c0b6114 2024-03-24T12:01:18.831 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (29s) 6s ago 19m 46.9M - 2.43.0 a07b618ecd1d 44f66fa22974 2024-03-24T12:01:19.245 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:01:19.245 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T12:01:19.245 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:01:19.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:19.245 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T12:01:19.246 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:01:19.594 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:01:19.594 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "nfs", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "18/23 daemons upgraded", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading grafana daemons", 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T12:01:19.595 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:01:19.739 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:19 smithi032 ceph-mon[66455]: pgmap v368: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:19.739 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:19 smithi032 ceph-mon[66455]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:19.739 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:19 smithi032 ceph-mon[66455]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:19.739 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:19 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/1739014913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:19.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:19 smithi146 ceph-mon[53008]: pgmap v368: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:19.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:19 smithi146 ceph-mon[53008]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:19.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:19 smithi146 ceph-mon[53008]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:19.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:19 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/1739014913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:20.010 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T12:01:20.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:20 smithi032 ceph-mon[66455]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:20.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:20 smithi032 ceph-mon[66455]: Upgrade: Updating grafana.smithi032 2024-03-24T12:01:20.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:20 smithi032 ceph-mon[66455]: Deploying daemon grafana.smithi032 on smithi032 2024-03-24T12:01:20.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:20 smithi032 ceph-mon[66455]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:20.702 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:20 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3318914540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:01:20.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:20 smithi146 ceph-mon[53008]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:20.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:20 smithi146 ceph-mon[53008]: Upgrade: Updating grafana.smithi032 2024-03-24T12:01:20.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:20 smithi146 ceph-mon[53008]: Deploying daemon grafana.smithi032 on smithi032 2024-03-24T12:01:20.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:20 smithi146 ceph-mon[53008]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:20.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:20 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3318914540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:01:21.685 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:21 smithi032 ceph-mon[66455]: pgmap v369: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:21.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:21 smithi146 ceph-mon[53008]: pgmap v369: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:23.761 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:23 smithi032 ceph-mon[66455]: pgmap v370: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:23.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:23 smithi146 ceph-mon[53008]: pgmap v370: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:24.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:24 smithi032 ceph-mon[66455]: pgmap v371: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:24.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:24.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:24.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:24.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:24.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:24 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:25.133 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:24 smithi146 ceph-mon[53008]: pgmap v371: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:25.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:25.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:25.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:25.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:25.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:24 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:26.122 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:25 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:26.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:25 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:27.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:26 smithi032 ceph-mon[66455]: pgmap v372: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:27.133 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:26 smithi146 ceph-mon[53008]: pgmap v372: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:29.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:28 smithi146 ceph-mon[53008]: pgmap v373: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:29.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:28 smithi032 ceph-mon[66455]: pgmap v373: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:30.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:30.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:29 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:30.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:30.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:29 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:31 smithi032 ceph-mon[66455]: pgmap v374: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:31 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:31.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:31 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:31.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:31 smithi146 ceph-mon[53008]: pgmap v374: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:31.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:31 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:31.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:31 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:33.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:33 smithi032 ceph-mon[66455]: pgmap v375: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:33.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:33 smithi146 ceph-mon[53008]: pgmap v375: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:34.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: pgmap v376: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.953 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.954 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: Upgrade: Finalizing container_image settings 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-24T12:01:34.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.956 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:34.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: Upgrade: Complete! 2024-03-24T12:01:34.957 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-24T12:01:34.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-24T12:01:34.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:34.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:34.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:34.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.958 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:34.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:34.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:34.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:34.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:34 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.134 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: pgmap v376: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.135 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.136 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: Upgrade: Finalizing container_image settings 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-24T12:01:35.137 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.138 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: Upgrade: Complete! 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.139 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:35.140 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:35.140 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:35.140 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:35.140 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:34 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:37.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:37 smithi146 ceph-mon[53008]: pgmap v377: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:37.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:37 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:37.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:37 smithi032 ceph-mon[66455]: pgmap v377: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:37.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:37 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:01:39.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:39 smithi146 ceph-mon[53008]: pgmap v378: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:39.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:39 smithi032 ceph-mon[66455]: pgmap v378: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:41.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:41 smithi146 ceph-mon[53008]: pgmap v379: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:41.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:41.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:41 smithi032 ceph-mon[66455]: pgmap v379: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:41.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:43 smithi146 ceph-mon[53008]: pgmap v380: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:43.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:43 smithi032 ceph-mon[66455]: pgmap v380: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:45.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:45 smithi146 ceph-mon[53008]: pgmap v381: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:45.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:45 smithi032 ceph-mon[66455]: pgmap v381: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:47.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:47 smithi146 ceph-mon[53008]: pgmap v382: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:47.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:47 smithi032 ceph-mon[66455]: pgmap v382: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:49.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:49 smithi146 ceph-mon[53008]: pgmap v383: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:49.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:49 smithi032 ceph-mon[66455]: pgmap v383: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:50.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch ps' 2024-03-24T12:01:51.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:51 smithi032 ceph-mon[66455]: pgmap v384: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:51.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:51 smithi032 ceph-mon[66455]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:51 smithi146 ceph-mon[53008]: pgmap v384: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:51.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:51 smithi146 ceph-mon[53008]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:52.941 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T12:01:52.941 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (46s) 23s ago 21m 15.9M - 0.25.0 c8568f914cd2 7e89a3c078b6 2024-03-24T12:01:52.941 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 23s ago 21m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T12:01:52.941 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (8m) 77s ago 20m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T12:01:52.941 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (29s) 23s ago 21m 56.1M - 9.4.12 a0ea5e3c46d9 c9c04bb5009a 2024-03-24T12:01:52.941 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (2m) 23s ago 15m 36.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 95c8d849ddcd 2024-03-24T12:01:52.941 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (2m) 77s ago 15m 11.5M - 19.0.0-1667-gdb0330b1 4f325f8440e9 976a8ba8b4ad 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (11m) 23s ago 22m 516M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (10m) 77s ago 20m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 23s ago 22m 63.5M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (9m) 77s ago 20m 50.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (111s) 23s ago 15m 62.4M - 5.7 4f325f8440e9 c8d78602083c 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (105s) 77s ago 15m 57.2M - 5.7 4f325f8440e9 21450c0428e3 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (87s) 23s ago 21m 8354k - 1.5.0 0da6a335fe13 f5d342c5eb41 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (81s) 77s ago 20m 3313k - 1.5.0 0da6a335fe13 63e203152582 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 23s ago 19m 432M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 23s ago 19m 397M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 23s ago 18m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (5m) 23s ago 18m 438M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (4m) 77s ago 18m 342M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T12:01:52.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (4m) 77s ago 17m 307M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T12:01:52.943 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (3m) 77s ago 17m 291M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T12:01:52.943 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (2m) 77s ago 16m 301M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 85b15c0b6114 2024-03-24T12:01:52.943 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (63s) 23s ago 20m 47.0M - 2.43.0 a07b618ecd1d 44f66fa22974 2024-03-24T12:01:53.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:53 smithi032 ceph-mon[66455]: pgmap v385: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:53.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:53 smithi146 ceph-mon[53008]: pgmap v385: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:53.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions' 2024-03-24T12:01:54.149 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:54 smithi032 ceph-mon[66455]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:54.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:54 smithi146 ceph-mon[53008]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:01:55.180 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:55 smithi032 ceph-mon[66455]: pgmap v386: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:55.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:55 smithi146 ceph-mon[53008]: pgmap v386: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:01:55.677 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T12:01:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T12:01:55.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T12:01:55.679 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:01:56.268 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:56 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/1405095720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:56.268 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:56.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-24T12:01:56.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:56 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/1405095720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:01:56.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:57.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:57 smithi032 ceph-mon[66455]: pgmap v387: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:57.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:57 smithi146 ceph-mon[53008]: pgmap v387: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:58.079 INFO:teuthology.orchestra.run.smithi032.stdout:wait for servicemap items w/ changing names to refresh 2024-03-24T12:01:58.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'sleep 60' 2024-03-24T12:01:59.353 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:01:59 smithi032 ceph-mon[66455]: pgmap v388: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:01:59.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:01:59 smithi146 ceph-mon[53008]: pgmap v388: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:01.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:01 smithi146 ceph-mon[53008]: pgmap v389: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:01.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:01 smithi032 ceph-mon[66455]: pgmap v389: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:03.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:03 smithi146 ceph-mon[53008]: pgmap v390: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:03.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:03 smithi032 ceph-mon[66455]: pgmap v390: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:05.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:05 smithi146 ceph-mon[53008]: pgmap v391: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:05.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:05 smithi032 ceph-mon[66455]: pgmap v391: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:07.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:07 smithi146 ceph-mon[53008]: pgmap v392: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:07.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:07 smithi032 ceph-mon[66455]: pgmap v392: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:09.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:09 smithi146 ceph-mon[53008]: pgmap v393: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:09.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:09 smithi032 ceph-mon[66455]: pgmap v393: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:11.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:11 smithi146 ceph-mon[53008]: pgmap v394: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:11.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:11.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:11 smithi032 ceph-mon[66455]: pgmap v394: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:11.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:13.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:13 smithi146 ceph-mon[53008]: pgmap v395: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:13.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:13 smithi032 ceph-mon[66455]: pgmap v395: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:15.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:15 smithi146 ceph-mon[53008]: pgmap v396: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:15.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:15 smithi032 ceph-mon[66455]: pgmap v396: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:17.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:17 smithi146 ceph-mon[53008]: pgmap v397: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:17.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:17 smithi032 ceph-mon[66455]: pgmap v397: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:19.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:19 smithi146 ceph-mon[53008]: pgmap v398: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:19.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:19 smithi032 ceph-mon[66455]: pgmap v398: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:21.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:21 smithi146 ceph-mon[53008]: pgmap v399: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:21.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:21 smithi032 ceph-mon[66455]: pgmap v399: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:23.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:23 smithi146 ceph-mon[53008]: pgmap v400: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:23.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:23 smithi032 ceph-mon[66455]: pgmap v400: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:25.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:25 smithi146 ceph-mon[53008]: pgmap v401: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:25.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:25 smithi032 ceph-mon[66455]: pgmap v401: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:26.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:26.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:27.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:27 smithi146 ceph-mon[53008]: pgmap v402: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:27.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:27 smithi032 ceph-mon[66455]: pgmap v402: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:29.385 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:29 smithi146 ceph-mon[53008]: pgmap v403: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:29.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:29 smithi032 ceph-mon[66455]: pgmap v403: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:31.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:31 smithi146 ceph-mon[53008]: pgmap v404: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:31.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:31 smithi032 ceph-mon[66455]: pgmap v404: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:33.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:33 smithi146 ceph-mon[53008]: pgmap v405: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:33.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:33 smithi032 ceph-mon[66455]: pgmap v405: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:35 smithi146 ceph-mon[53008]: pgmap v406: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:02:35.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:35 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:02:35.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:35 smithi032 ceph-mon[66455]: pgmap v406: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:35.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:35.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:35.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:35.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:02:35.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:35 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' 2024-03-24T12:02:37.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:37 smithi146 ceph-mon[53008]: pgmap v407: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:37.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:37 smithi032 ceph-mon[66455]: pgmap v407: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:39.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:39 smithi146 ceph-mon[53008]: pgmap v408: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:39.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:39 smithi032 ceph-mon[66455]: pgmap v408: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:41.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:41 smithi146 ceph-mon[53008]: pgmap v409: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:41.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:41 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:41.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:41 smithi032 ceph-mon[66455]: pgmap v409: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:41.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:41 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:43.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:43 smithi146 ceph-mon[53008]: pgmap v410: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:43.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:43 smithi032 ceph-mon[66455]: pgmap v410: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:45.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:45 smithi146 ceph-mon[53008]: pgmap v411: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:45.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:45 smithi032 ceph-mon[66455]: pgmap v411: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:47.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:47 smithi146 ceph-mon[53008]: pgmap v412: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:47.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:47 smithi032 ceph-mon[66455]: pgmap v412: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:49.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:49 smithi032 ceph-mon[66455]: pgmap v413: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:49.633 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:49 smithi146 ceph-mon[53008]: pgmap v413: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:51.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:51 smithi032 ceph-mon[66455]: pgmap v414: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:51.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:51 smithi146 ceph-mon[53008]: pgmap v414: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:53.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:53 smithi032 ceph-mon[66455]: pgmap v415: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:53.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:53 smithi146 ceph-mon[53008]: pgmap v415: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:55.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:55 smithi032 ceph-mon[66455]: pgmap v416: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:55.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:55 smithi146 ceph-mon[53008]: pgmap v416: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:02:56.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:56 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:56.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:56 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:57.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:57 smithi032 ceph-mon[66455]: pgmap v417: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:57.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:57 smithi146 ceph-mon[53008]: pgmap v417: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:59.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:02:59 smithi032 ceph-mon[66455]: pgmap v418: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:02:59.633 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:02:59 smithi146 ceph-mon[53008]: pgmap v418: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:01.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch ps' 2024-03-24T12:03:01.416 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:01 smithi032 ceph-mon[66455]: pgmap v419: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:01.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:01 smithi146 ceph-mon[53008]: pgmap v419: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T12:03:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (116s) 94s ago 22m 15.9M - 0.25.0 c8568f914cd2 7e89a3c078b6 2024-03-24T12:03:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 94s ago 22m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 d77c5c4fe4fb 2024-03-24T12:03:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi146 smithi146 running (9m) 2m ago 21m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 b4ae9e66331c 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (99s) 94s ago 22m 56.1M - 9.4.12 a0ea5e3c46d9 c9c04bb5009a 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi032.qzdcjy smithi032 running (3m) 94s ago 16m 36.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 95c8d849ddcd 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:mds.foofs.smithi146.xsrsue smithi146 running (3m) 2m ago 16m 11.5M - 19.0.0-1667-gdb0330b1 4f325f8440e9 976a8ba8b4ad 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.xjdgbw smithi032 *:8443,9283,8765 running (12m) 94s ago 23m 516M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fc6fe5231ca9 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi146.tfohfl smithi146 *:8443,9283,8765 running (11m) 2m ago 21m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 4dd5f6e8f033 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 94s ago 23m 63.5M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 c8989f11d4c0 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi146 smithi146 running (10m) 2m ago 21m 50.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 61b9704b5f29 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.0.0.smithi032.lizmvs smithi032 *:2049 running (3m) 94s ago 16m 62.4M - 5.7 4f325f8440e9 c8d78602083c 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:nfs.foo.1.0.smithi146.rjcwds smithi146 *:2049 running (2m) 2m ago 16m 57.2M - 5.7 4f325f8440e9 21450c0428e3 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (2m) 94s ago 22m 8354k - 1.5.0 0da6a335fe13 f5d342c5eb41 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi146 smithi146 *:9100 running (2m) 2m ago 21m 3313k - 1.5.0 0da6a335fe13 63e203152582 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 94s ago 21m 432M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ec0013a993c 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 94s ago 20m 397M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 6e2b66b3ab3a 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 94s ago 20m 355M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 ab8a1998f9b7 2024-03-24T12:03:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (6m) 94s ago 19m 438M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 05e7bad94028 2024-03-24T12:03:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi146 running (6m) 2m ago 19m 342M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 84c8e53a61a8 2024-03-24T12:03:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi146 running (5m) 2m ago 18m 307M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 3ce00834b206 2024-03-24T12:03:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi146 running (4m) 2m ago 18m 291M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 8bab439aede4 2024-03-24T12:03:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi146 running (4m) 2m ago 18m 301M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 85b15c0b6114 2024-03-24T12:03:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (2m) 94s ago 21m 47.0M - 2.43.0 a07b618ecd1d 44f66fa22974 2024-03-24T12:03:03.451 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:03 smithi032 ceph-mon[66455]: pgmap v420: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:03.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:03 smithi146 ceph-mon[53008]: pgmap v420: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:03.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch upgrade status' 2024-03-24T12:03:04.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:04 smithi032 ceph-mon[66455]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:04.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:04 smithi146 ceph-mon[53008]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:05.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:05 smithi032 ceph-mon[66455]: pgmap v421: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:03:05.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:05 smithi146 ceph-mon[53008]: pgmap v421: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:03:05.800 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:03:05.800 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": null, 2024-03-24T12:03:05.800 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": false, 2024-03-24T12:03:05.800 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "", 2024-03-24T12:03:05.800 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-03-24T12:03:05.800 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": null, 2024-03-24T12:03:05.801 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-03-24T12:03:05.801 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-03-24T12:03:05.801 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:03:06.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph health detail' 2024-03-24T12:03:07.348 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:07 smithi032 ceph-mon[66455]: pgmap v422: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:07.348 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:07 smithi032 ceph-mon[66455]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:07.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:07 smithi146 ceph-mon[53008]: pgmap v422: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:07.634 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:07 smithi146 ceph-mon[53008]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:08.516 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-03-24T12:03:08.823 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:08 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/695237025' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:03:08.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:08 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/695237025' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T12:03:09.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions' 2024-03-24T12:03:09.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:09 smithi032 ceph-mon[66455]: pgmap v423: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:09.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:09 smithi146 ceph-mon[53008]: pgmap v423: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:11.231 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-24T12:03:11.231 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-03-24T12:03:11.231 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T12:03:11.232 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-24T12:03:11.879 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:11 smithi032 ceph-mon[66455]: pgmap v424: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:11.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:11 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:11.880 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:11 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/3017863074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:03:11.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:11 smithi146 ceph-mon[53008]: pgmap v424: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:11 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:11.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:11 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/3017863074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:03:11.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-24T12:03:13.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:13 smithi032 ceph-mon[66455]: pgmap v425: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:13.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:13 smithi146 ceph-mon[53008]: pgmap v425: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:14.039 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-03-24T12:03:14.597 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:14 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2975139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:03:14.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-24T12:03:14.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:14 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/2975139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:03:15.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:15 smithi146 ceph-mon[53008]: pgmap v426: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:03:15.917 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:15 smithi032 ceph-mon[66455]: pgmap v426: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:03:16.696 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)" 2024-03-24T12:03:17.311 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-03-24T12:03:17.312 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-03-24T12:03:17.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T12:03:17.870 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:17 smithi032 ceph-mon[66455]: pgmap v427: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:17.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:17 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/136526384' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:03:17.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:17 smithi146 ceph-mon[53008]: pgmap v427: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:17.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:17 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/136526384' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T12:03:19.372 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-24T12:03:19.373 INFO:teuthology.orchestra.run.smithi032.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T11:39:55.322739Z", "last_refresh": "2024-03-24T12:01:28.958991Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T11:39:49.097615Z", "last_refresh": "2024-03-24T12:00:35.332520Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-03-24T11:39:52.284325Z", "last_refresh": "2024-03-24T12:01:28.959169Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T11:46:17.041029Z", "last_refresh": "2024-03-24T12:00:35.333196Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T11:39:47.601685Z", "last_refresh": "2024-03-24T12:00:35.332625Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi032:172.21.15.32=smithi032", "smithi146:172.21.15.146=smithi146"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T11:40:37.521788Z", "last_refresh": "2024-03-24T12:00:35.332708Z", "running": 2, "size": 2}}, {"events": ["2024-03-24T12:02:34.781634Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-03-24T11:46:36.843676Z", "last_refresh": "2024-03-24T12:00:35.333277Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T11:39:53.434504Z", "last_refresh": "2024-03-24T12:00:35.332788Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", "last_refresh": "2024-03-24T12:00:35.332867Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T11:39:50.497551Z", "last_refresh": "2024-03-24T12:01:28.959340Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T12:03:19.664 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:19 smithi032 ceph-mon[66455]: pgmap v428: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:19.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:19 smithi146 ceph-mon[53008]: pgmap v428: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:19.980 INFO:tasks.cephadm:nfs.foo has 2/2 2024-03-24T12:03:19.981 DEBUG:teuthology.parallel:result is None 2024-03-24T12:03:19.981 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-24T12:03:19.991 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T12:03:19.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-03-24T12:03:20.020 INFO:teuthology.orchestra.run.smithi032.stderr:+ umount /mnt/foo 2024-03-24T12:03:20.031 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T12:03:20.041 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T12:03:20.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-03-24T12:03:20.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:20 smithi032 ceph-mon[66455]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:03:20.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:20 smithi146 ceph-mon[53008]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:03:21.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:21 smithi146 ceph-mon[53008]: pgmap v429: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:21.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:21 smithi032 ceph-mon[66455]: pgmap v429: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:22.042 INFO:teuthology.orchestra.run.smithi032.stdout: "foo" 2024-03-24T12:03:22.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs export ls foo --detailed' 2024-03-24T12:03:23.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:23 smithi146 ceph-mon[53008]: pgmap v430: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:23.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:23 smithi146 ceph-mon[53008]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:23.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:23 smithi032 ceph-mon[66455]: pgmap v430: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:23.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:23 smithi032 ceph-mon[66455]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:24.557 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-24T12:03:24.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-24T12:03:24.557 INFO:teuthology.orchestra.run.smithi032.stdout: "access_type": "RW", 2024-03-24T12:03:24.557 INFO:teuthology.orchestra.run.smithi032.stdout: "clients": [], 2024-03-24T12:03:24.557 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_id": "foo", 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "export_id": 1, 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "fsal": { 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "foofs", 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "CEPH", 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "user_id": "nfs.foo.1" 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/", 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "protocols": [ 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: 4 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "pseudo": "/fake", 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "security_label": true, 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "squash": "none", 2024-03-24T12:03:24.558 INFO:teuthology.orchestra.run.smithi032.stdout: "transports": [ 2024-03-24T12:03:24.559 INFO:teuthology.orchestra.run.smithi032.stdout: "TCP" 2024-03-24T12:03:24.559 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-24T12:03:24.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-24T12:03:24.559 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-24T12:03:25.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'rados -p .nfs --all ls -' 2024-03-24T12:03:25.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:25 smithi032 ceph-mon[66455]: pgmap v431: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:03:25.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:25 smithi032 ceph-mon[66455]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:25.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:25 smithi032 ceph-mon[66455]: mgrmap e42: smithi032.xjdgbw(active, since 11m), standbys: smithi146.tfohfl 2024-03-24T12:03:25.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:25 smithi146 ceph-mon[53008]: pgmap v431: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T12:03:25.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:25 smithi146 ceph-mon[53008]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:03:25.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:25 smithi146 ceph-mon[53008]: mgrmap e42: smithi032.xjdgbw(active, since 11m), standbys: smithi146.tfohfl 2024-03-24T12:03:26.696 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:26 smithi032 ceph-mon[66455]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:26.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:26 smithi146 ceph-mon[53008]: from='mgr.34102 172.21.15.32:0/1303406563' entity='mgr.smithi032.xjdgbw' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:26.905 INFO:teuthology.orchestra.run.smithi032.stdout:foo rec-0000000000000004:nfs.foo.1 2024-03-24T12:03:26.905 INFO:teuthology.orchestra.run.smithi032.stdout:foo grace 2024-03-24T12:03:26.905 INFO:teuthology.orchestra.run.smithi032.stdout:foo rec-0000000000000004:nfs.foo.0 2024-03-24T12:03:26.905 INFO:teuthology.orchestra.run.smithi032.stdout:foo conf-nfs.foo 2024-03-24T12:03:26.905 INFO:teuthology.orchestra.run.smithi032.stdout:foo export-1 2024-03-24T12:03:27.488 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T12:03:27.498 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T12:03:27.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 -- bash -c 'set -ex 2024-03-24T12:03:27.498 DEBUG:teuthology.orchestra.run.smithi032:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-03-24T12:03:27.498 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-24T12:03:27.816 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:27 smithi032 ceph-mon[66455]: pgmap v432: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:27.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:27 smithi146 ceph-mon[53008]: pgmap v432: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-03-24T12:03:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:+ [[ 6 -gt 2 ]] 2024-03-24T12:03:29.701 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:29 smithi032 ceph-mon[66455]: pgmap v433: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:29.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:29 smithi146 ceph-mon[53008]: pgmap v433: 97 pgs: 97 active+clean; 233 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T12:03:30.275 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T12:03:30.285 INFO:tasks.cephadm:Teardown begin 2024-03-24T12:03:30.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:03:30.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:03:30.352 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T12:03:30.352 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:03:30.371 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:03:30.409 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T12:03:30.409 INFO:tasks.cephadm.mon.smithi032:Stopping mon.smithi032... 2024-03-24T12:03:30.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032 2024-03-24T12:03:30.609 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:30 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:03:30.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:30 smithi146 ceph-mon[53008]: from='client.? 172.21.15.32:0/2959547699' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-24T12:03:30.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:30 smithi032 ceph-mon[66455]: from='client.? 172.21.15.32:0/2959547699' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-24T12:03:30.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:30 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[66451]: 2024-03-24T12:03:30.702+0000 7efe7a8b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:03:30.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:30 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032[66451]: 2024-03-24T12:03:30.702+0000 7efe7a8b5700 -1 mon.smithi032@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T12:03:30.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:30 smithi032 podman[111404]: 2024-03-24 12:03:30.724351761 +0000 UTC m=+0.122942214 container died c8989f11d4c04750b94923e5a91fc6d640e853710c9d6ac377288a87077494d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T12:03:31.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:31 smithi032 podman[111404]: 2024-03-24 12:03:31.11204393 +0000 UTC m=+0.510634379 container cleanup c8989f11d4c04750b94923e5a91fc6d640e853710c9d6ac377288a87077494d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:03:31.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Mar 24 12:03:31 smithi032 bash[111404]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi032 2024-03-24T12:03:31.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi032.service' 2024-03-24T12:03:31.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:03:31.540 INFO:tasks.cephadm.mon.smithi032:Stopped mon.smithi032 2024-03-24T12:03:31.541 INFO:tasks.cephadm.mon.smithi146:Stopping mon.smithi146... 2024-03-24T12:03:31.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi146 2024-03-24T12:03:31.883 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:31 smithi146 systemd[1]: Stopping Ceph mon.smithi146 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:03:32.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:31 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146[53004]: 2024-03-24T12:03:31.996+0000 7f5292fb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi146 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:03:32.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:31 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146[53004]: 2024-03-24T12:03:31.996+0000 7f5292fb6700 -1 mon.smithi146@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T12:03:32.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:32 smithi146 podman[78339]: 2024-03-24 12:03:32.231462445 +0000 UTC m=+0.489779969 container died 61b9704b5f2997542df34723c0a48ebe59b20dd548343dae4486db677a67bda4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b) 2024-03-24T12:03:32.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:32 smithi146 podman[78339]: 2024-03-24 12:03:32.713291989 +0000 UTC m=+0.971609564 container cleanup 61b9704b5f2997542df34723c0a48ebe59b20dd548343dae4486db677a67bda4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:03:32.884 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:Mar 24 12:03:32 smithi146 bash[78339]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-mon-smithi146 2024-03-24T12:03:33.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@mon.smithi146.service' 2024-03-24T12:03:33.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:03:33.222 INFO:tasks.cephadm.mon.smithi146:Stopped mon.smithi146 2024-03-24T12:03:33.222 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T12:03:33.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.0 2024-03-24T12:03:33.543 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:33 smithi032 systemd[1]: Stopping Ceph osd.0 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:03:33.951 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:33 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[72577]: 2024-03-24T12:03:33.544+0000 7fe2a2bbd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:03:33.952 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:33 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[72577]: 2024-03-24T12:03:33.544+0000 7fe2a2bbd700 -1 osd.0 156 *** Got signal Terminated *** 2024-03-24T12:03:33.952 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:33 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0[72577]: 2024-03-24T12:03:33.544+0000 7fe2a2bbd700 -1 osd.0 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:03:39.235 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:38 smithi032 podman[111571]: 2024-03-24 12:03:38.957122736 +0000 UTC m=+5.541333472 container died 7ec0013a993cc22681fee213e0e548c539e0b29bf85297130ff21e08d95db5f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:03:39.585 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:39 smithi032 podman[111571]: 2024-03-24 12:03:39.32720825 +0000 UTC m=+5.911418983 container cleanup 7ec0013a993cc22681fee213e0e548c539e0b29bf85297130ff21e08d95db5f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:03:39.586 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:39 smithi032 bash[111571]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0 2024-03-24T12:03:39.951 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:39 smithi032 podman[111588]: 2024-03-24 12:03:39.586951027 +0000 UTC m=+0.628025080 container remove 7ec0013a993cc22681fee213e0e548c539e0b29bf85297130ff21e08d95db5f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:03:40.223 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:39 smithi032 podman[111684]: 2024-03-24 12:03:39.859360438 +0000 UTC m=+0.018939999 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:03:40.223 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:39 smithi032 podman[111684]: 2024-03-24 12:03:39.998348855 +0000 UTC m=+0.157928404 container create 3dbf751dbf804674ef9f93cb22a2b95510b80cb6dd8ce1910a5013bcd03a9efb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T12:03:40.223 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:40 smithi032 podman[111684]: 2024-03-24 12:03:40.224330667 +0000 UTC m=+0.383910231 container init 3dbf751dbf804674ef9f93cb22a2b95510b80cb6dd8ce1910a5013bcd03a9efb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T12:03:40.519 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:40 smithi032 podman[111684]: 2024-03-24 12:03:40.228878982 +0000 UTC m=+0.388458531 container start 3dbf751dbf804674ef9f93cb22a2b95510b80cb6dd8ce1910a5013bcd03a9efb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:03:40.519 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:40 smithi032 podman[111684]: 2024-03-24 12:03:40.299163742 +0000 UTC m=+0.458743307 container attach 3dbf751dbf804674ef9f93cb22a2b95510b80cb6dd8ce1910a5013bcd03a9efb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T12:03:40.868 INFO:journalctl@ceph.osd.0.smithi032.stdout:Mar 24 12:03:40 smithi032 podman[111684]: 2024-03-24 12:03:40.520074531 +0000 UTC m=+0.679654073 container died 3dbf751dbf804674ef9f93cb22a2b95510b80cb6dd8ce1910a5013bcd03a9efb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:03:41.151 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T12:03:41.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:03:41.193 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T12:03:41.193 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T12:03:41.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.1 2024-03-24T12:03:41.452 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:41 smithi032 systemd[1]: Stopping Ceph osd.1 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:03:41.951 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:41 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[75170]: 2024-03-24T12:03:41.539+0000 7f0fe6d8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:03:41.951 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:41 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[75170]: 2024-03-24T12:03:41.539+0000 7f0fe6d8d700 -1 osd.1 156 *** Got signal Terminated *** 2024-03-24T12:03:41.952 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:41 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1[75170]: 2024-03-24T12:03:41.539+0000 7f0fe6d8d700 -1 osd.1 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:03:47.357 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:47 smithi032 podman[111790]: 2024-03-24 12:03:47.078738281 +0000 UTC m=+5.659299645 container died 6e2b66b3ab3a8bae53ce7404cf562f1bf3547ef678115d6ccbcbe998cac297de (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T12:03:47.701 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:47 smithi032 podman[111790]: 2024-03-24 12:03:47.431986716 +0000 UTC m=+6.012548097 container cleanup 6e2b66b3ab3a8bae53ce7404cf562f1bf3547ef678115d6ccbcbe998cac297de (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T12:03:47.701 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:47 smithi032 bash[111790]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1 2024-03-24T12:03:48.058 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:47 smithi032 podman[111807]: 2024-03-24 12:03:47.724845333 +0000 UTC m=+0.640432667 container remove 6e2b66b3ab3a8bae53ce7404cf562f1bf3547ef678115d6ccbcbe998cac297de (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T12:03:48.394 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:48 smithi032 podman[111901]: 2024-03-24 12:03:48.059271095 +0000 UTC m=+0.115375724 container create 9f4954912fc989fb2ad1daf9696fd5b8cc78a466eae1e69c1d4ee5ae1a3df5ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T12:03:48.394 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:48 smithi032 podman[111901]: 2024-03-24 12:03:47.963017199 +0000 UTC m=+0.019121795 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:03:48.394 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:48 smithi032 podman[111901]: 2024-03-24 12:03:48.295183825 +0000 UTC m=+0.351288418 container init 9f4954912fc989fb2ad1daf9696fd5b8cc78a466eae1e69c1d4ee5ae1a3df5ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T12:03:48.394 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:48 smithi032 podman[111901]: 2024-03-24 12:03:48.298561943 +0000 UTC m=+0.354666530 container start 9f4954912fc989fb2ad1daf9696fd5b8cc78a466eae1e69c1d4ee5ae1a3df5ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:03:48.394 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:48 smithi032 podman[111901]: 2024-03-24 12:03:48.395167928 +0000 UTC m=+0.451272534 container attach 9f4954912fc989fb2ad1daf9696fd5b8cc78a466eae1e69c1d4ee5ae1a3df5ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T12:03:48.702 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:48 smithi032 podman[111901]: 2024-03-24 12:03:48.593624253 +0000 UTC m=+0.649728843 container died 9f4954912fc989fb2ad1daf9696fd5b8cc78a466eae1e69c1d4ee5ae1a3df5ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b) 2024-03-24T12:03:49.159 INFO:journalctl@ceph.osd.1.smithi032.stdout:Mar 24 12:03:49 smithi032 podman[111928]: 2024-03-24 12:03:49.157539608 +0000 UTC m=+0.554549460 container remove 9f4954912fc989fb2ad1daf9696fd5b8cc78a466eae1e69c1d4ee5ae1a3df5ad (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:03:49.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T12:03:49.211 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:03:49.212 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T12:03:49.212 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T12:03:49.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.2 2024-03-24T12:03:49.451 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:49 smithi032 systemd[1]: Stopping Ceph osd.2 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:03:49.951 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:49 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[77867]: 2024-03-24T12:03:49.560+0000 7f4e45623700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:03:49.952 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:49 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[77867]: 2024-03-24T12:03:49.560+0000 7f4e45623700 -1 osd.2 156 *** Got signal Terminated *** 2024-03-24T12:03:49.952 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:49 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2[77867]: 2024-03-24T12:03:49.560+0000 7f4e45623700 -1 osd.2 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:03:55.386 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:55 smithi032 podman[112008]: 2024-03-24 12:03:55.130238578 +0000 UTC m=+5.695698927 container died ab8a1998f9b73f98d6f4cf61a2976c7c3569848f10a0990164d8ee1e26f9950f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T12:03:55.701 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:55 smithi032 podman[112008]: 2024-03-24 12:03:55.477285197 +0000 UTC m=+6.042745550 container cleanup ab8a1998f9b73f98d6f4cf61a2976c7c3569848f10a0990164d8ee1e26f9950f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T12:03:55.702 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:55 smithi032 bash[112008]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2 2024-03-24T12:03:56.093 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:55 smithi032 podman[112057]: 2024-03-24 12:03:55.770077481 +0000 UTC m=+0.639073860 container remove ab8a1998f9b73f98d6f4cf61a2976c7c3569848f10a0990164d8ee1e26f9950f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T12:03:56.381 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:56 smithi032 podman[112149]: 2024-03-24 12:03:55.997938585 +0000 UTC m=+0.019329732 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:03:56.381 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:56 smithi032 podman[112149]: 2024-03-24 12:03:56.112695157 +0000 UTC m=+0.134086272 container create 1884c007f3684aadccd126d1222e9d039030e2397949fbb74a984c830cb1a23d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:03:56.381 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:56 smithi032 podman[112149]: 2024-03-24 12:03:56.382376649 +0000 UTC m=+0.403767786 container init 1884c007f3684aadccd126d1222e9d039030e2397949fbb74a984c830cb1a23d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:03:56.682 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:56 smithi032 podman[112149]: 2024-03-24 12:03:56.386280908 +0000 UTC m=+0.407672025 container start 1884c007f3684aadccd126d1222e9d039030e2397949fbb74a984c830cb1a23d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:03:56.682 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:56 smithi032 podman[112149]: 2024-03-24 12:03:56.457312252 +0000 UTC m=+0.478703368 container attach 1884c007f3684aadccd126d1222e9d039030e2397949fbb74a984c830cb1a23d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T12:03:56.951 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:56 smithi032 podman[112149]: 2024-03-24 12:03:56.683292265 +0000 UTC m=+0.704683386 container died 1884c007f3684aadccd126d1222e9d039030e2397949fbb74a984c830cb1a23d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T12:03:57.261 INFO:journalctl@ceph.osd.2.smithi032.stdout:Mar 24 12:03:57 smithi032 podman[112176]: 2024-03-24 12:03:57.260438786 +0000 UTC m=+0.571489510 container remove 1884c007f3684aadccd126d1222e9d039030e2397949fbb74a984c830cb1a23d (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T12:03:57.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T12:03:57.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:03:57.325 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T12:03:57.325 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T12:03:57.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.3 2024-03-24T12:03:57.671 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:03:57 smithi032 systemd[1]: Stopping Ceph osd.3 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:03:57.951 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:03:57 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T12:03:57.671+0000 7ff1027ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:03:57.952 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:03:57 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T12:03:57.671+0000 7ff1027ab700 -1 osd.3 156 *** Got signal Terminated *** 2024-03-24T12:03:57.952 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:03:57 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T12:03:57.671+0000 7ff1027ab700 -1 osd.3 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:04:00.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:00 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:00.224+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:00.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:00 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:00.299+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:01.569 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:01 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:01.244+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:01.570 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:01 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:01.332+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:01.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:01 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:01.570+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:01.951 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:01 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T12:04:01.538+0000 7ff0fad97700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.719925+0000 front 2024-03-24T12:03:36.720036+0000 (oldest deadline 2024-03-24T12:04:01.420036+0000) 2024-03-24T12:04:02.555 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:02 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:02.209+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:02.556 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:02 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:02.336+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:02.885 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:02 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:02.555+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:02.951 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:02 smithi032 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3[80468]: 2024-03-24T12:04:02.565+0000 7ff0fad97700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.719925+0000 front 2024-03-24T12:03:36.720036+0000 (oldest deadline 2024-03-24T12:04:01.420036+0000) 2024-03-24T12:04:03.556 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:03 smithi032 podman[112255]: 2024-03-24 12:04:03.267684483 +0000 UTC m=+5.717675457 container died 05e7bad940288770a4acfec0aa0d381614d3bad272ee2a97ea06d252671f416b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T12:04:03.585 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:03 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:03.433+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:03.585 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:03 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:03.184+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:03.586 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:03 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:03.384+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:03.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:03 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:03.584+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:03.947 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:03 smithi032 podman[112255]: 2024-03-24 12:04:03.63031072 +0000 UTC m=+6.080301702 container cleanup 05e7bad940288770a4acfec0aa0d381614d3bad272ee2a97ea06d252671f416b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T12:04:03.947 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:03 smithi032 bash[112255]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3 2024-03-24T12:04:04.201 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:03 smithi032 podman[112273]: 2024-03-24 12:04:03.948077779 +0000 UTC m=+0.671622209 container remove 05e7bad940288770a4acfec0aa0d381614d3bad272ee2a97ea06d252671f416b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T12:04:04.542 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:04 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:04.226+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:04.542 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:04 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:04.372+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:04.543 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:04 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:04.457+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:04.550 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:04 smithi032 podman[112369]: 2024-03-24 12:04:04.203200914 +0000 UTC m=+0.019846671 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:04:04.550 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:04 smithi032 podman[112369]: 2024-03-24 12:04:04.315612336 +0000 UTC m=+0.132258079 container create ae73aaf3dfdc21bc9d3ca6c165723218f32638ec7de5cbe898a2b213bd2b9c2f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:04:04.550 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:04 smithi032 podman[112369]: 2024-03-24 12:04:04.551408205 +0000 UTC m=+0.368053943 container init ae73aaf3dfdc21bc9d3ca6c165723218f32638ec7de5cbe898a2b213bd2b9c2f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T12:04:04.857 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:04 smithi032 podman[112369]: 2024-03-24 12:04:04.554700432 +0000 UTC m=+0.371346172 container start ae73aaf3dfdc21bc9d3ca6c165723218f32638ec7de5cbe898a2b213bd2b9c2f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:04:04.858 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:04 smithi032 podman[112369]: 2024-03-24 12:04:04.626318196 +0000 UTC m=+0.442963951 container attach ae73aaf3dfdc21bc9d3ca6c165723218f32638ec7de5cbe898a2b213bd2b9c2f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:04:04.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:04 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:04.541+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:05.195 INFO:journalctl@ceph.osd.3.smithi032.stdout:Mar 24 12:04:04 smithi032 podman[112369]: 2024-03-24 12:04:04.858826742 +0000 UTC m=+0.675472480 container died ae73aaf3dfdc21bc9d3ca6c165723218f32638ec7de5cbe898a2b213bd2b9c2f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:04:05.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T12:04:05.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:04:05.555 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T12:04:05.555 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T12:04:05.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.4 2024-03-24T12:04:05.571 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:05 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:05.275+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:05.572 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:05 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:05.358+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:05.572 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:05 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:05.463+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:05.884 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:05 smithi146 systemd[1]: Stopping Ceph osd.4 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:04:05.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:05 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:05.571+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:06.262 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:05 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:05.905+0000 7fbb107bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:04:06.262 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:05 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:05.905+0000 7fbb107bd700 -1 osd.4 156 *** Got signal Terminated *** 2024-03-24T12:04:06.262 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:05 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:05.905+0000 7fbb107bd700 -1 osd.4 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:04:06.262 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:06 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:06.261+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:06.556 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:06 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:06.317+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:06.556 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:06 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:06.424+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:06.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:06 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:06.556+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:07.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:07 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:07.299+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:07.634 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:07 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:07.390+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:07.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:07 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:07.324+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:07.634 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:07 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:07.520+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:08.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:08 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:08.333+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:08.634 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:08 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:08.490+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:08.634 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:08 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:08.257+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:08.635 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:08 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:08.386+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:08.635 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:08 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:08.386+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:09.536 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:09.289+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:09.537 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:09.238+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:09.537 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:09.238+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.783794+0000 front 2024-03-24T12:03:45.783817+0000 (oldest deadline 2024-03-24T12:04:08.683728+0000) 2024-03-24T12:04:09.537 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:09.384+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:09.537 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:09.384+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:09.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:09 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:09.536+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:10.571 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:10.256+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:32.981887+0000 front 2024-03-24T12:03:32.981692+0000 (oldest deadline 2024-03-24T12:03:59.982301+0000) 2024-03-24T12:04:10.572 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4[57944]: 2024-03-24T12:04:10.256+0000 7fbb08da9700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.783794+0000 front 2024-03-24T12:03:45.783817+0000 (oldest deadline 2024-03-24T12:04:08.683728+0000) 2024-03-24T12:04:10.572 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:10.277+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:10.572 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:10.277+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:10.572 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:10.422+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:10.572 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:10.422+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:10.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:10 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:10.572+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:11.384 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:11 smithi146 podman[78502]: 2024-03-24 12:04:11.271728587 +0000 UTC m=+5.512888035 container died 84c8e53a61a864543b833873eef52e76346f8dce868959299c61959c03e6e13b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T12:04:11.384 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:11.295+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:11.385 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:11.295+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:11.755 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:11.458+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:11.755 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:11.458+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:11.756 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:11.592+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:11.756 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:11 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:11.592+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:12.032 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:11 smithi146 podman[78502]: 2024-03-24 12:04:11.756881734 +0000 UTC m=+5.998041196 container cleanup 84c8e53a61a864543b833873eef52e76346f8dce868959299c61959c03e6e13b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T12:04:12.032 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:11 smithi146 bash[78502]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4 2024-03-24T12:04:12.350 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:12 smithi146 podman[78519]: 2024-03-24 12:04:12.033683792 +0000 UTC m=+0.762032665 container remove 84c8e53a61a864543b833873eef52e76346f8dce868959299c61959c03e6e13b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T12:04:12.350 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:12 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:12.271+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:12.350 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:12 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:12.271+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:12.602 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:12 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:12.474+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:12.602 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:12 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:12.474+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:12.602 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:12 smithi146 podman[78645]: 2024-03-24 12:04:12.351084595 +0000 UTC m=+0.101671776 container create 3eab4508d54eba8f51f89b065f0ecd668061e012d8c63229001ce3a984d78e29 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T12:04:12.603 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:12 smithi146 podman[78645]: 2024-03-24 12:04:12.26773568 +0000 UTC m=+0.018322820 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:04:12.603 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:12 smithi146 podman[78645]: 2024-03-24 12:04:12.579168783 +0000 UTC m=+0.329755920 container init 3eab4508d54eba8f51f89b065f0ecd668061e012d8c63229001ce3a984d78e29 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T12:04:12.603 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:12 smithi146 podman[78645]: 2024-03-24 12:04:12.582522547 +0000 UTC m=+0.333109686 container start 3eab4508d54eba8f51f89b065f0ecd668061e012d8c63229001ce3a984d78e29 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:04:12.872 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:12 smithi146 podman[78645]: 2024-03-24 12:04:12.678591245 +0000 UTC m=+0.429178381 container attach 3eab4508d54eba8f51f89b065f0ecd668061e012d8c63229001ce3a984d78e29 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T12:04:12.872 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:12 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:12.601+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:12.872 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:12 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:12.601+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:13.134 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 24 12:04:12 smithi146 podman[78645]: 2024-03-24 12:04:12.873809366 +0000 UTC m=+0.624396495 container died 3eab4508d54eba8f51f89b065f0ecd668061e012d8c63229001ce3a984d78e29 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-24T12:04:13.430 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:13.253+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:13.431 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:13.253+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:13.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T12:04:13.543 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:04:13.544 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T12:04:13.544 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T12:04:13.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:04:13.699 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:13 smithi146 systemd[1]: Stopping Ceph osd.5 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:04:13.700 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:13.621+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:13.700 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:13.621+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:13.700 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:13.464+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:13.700 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:13.464+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:14.134 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:13.892+0000 7f63eac01700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:04:14.134 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:13.892+0000 7f63eac01700 -1 osd.5 156 *** Got signal Terminated *** 2024-03-24T12:04:14.134 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:13 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:13.892+0000 7f63eac01700 -1 osd.5 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:04:14.607 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:14 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:14.244+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:14.607 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:14 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:14.244+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:14.607 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:14 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:14.459+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:14.608 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:14 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:14.459+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:14.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:14 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:14.607+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:14.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:14 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:14.607+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:15.589 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:15 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:15.252+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:15.589 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:15 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:15.252+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:15.590 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:15 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:15.471+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:15.590 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:15 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:15.471+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:15.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:15 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:15.590+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:15.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:15 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:15.590+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:16.621 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:16.256+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:16.621 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:16.256+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:16.621 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:16.256+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:49.498691+0000 front 2024-03-24T12:03:49.498710+0000 (oldest deadline 2024-03-24T12:04:15.398562+0000) 2024-03-24T12:04:16.621 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:16.482+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:16.622 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:16.482+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:16.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:16.621+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:16.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:16.621+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:16.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:16 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:16.621+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:17.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:17.257+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:17.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:17.257+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:17.634 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:17.257+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:49.498691+0000 front 2024-03-24T12:03:49.498710+0000 (oldest deadline 2024-03-24T12:04:15.398562+0000) 2024-03-24T12:04:17.634 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:17.483+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:17.634 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:17.483+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:17.660+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:17.660+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:18.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:17 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:17.660+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:18.629 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:18.268+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.596046+0000 front 2024-03-24T12:03:35.596161+0000 (oldest deadline 2024-03-24T12:03:59.696098+0000) 2024-03-24T12:04:18.629 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:18.268+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:41.897749+0000 front 2024-03-24T12:03:41.897680+0000 (oldest deadline 2024-03-24T12:04:09.498098+0000) 2024-03-24T12:04:18.629 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5[60222]: 2024-03-24T12:04:18.268+0000 7f63e31ed700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:49.498691+0000 front 2024-03-24T12:03:49.498710+0000 (oldest deadline 2024-03-24T12:04:15.398562+0000) 2024-03-24T12:04:18.629 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:18.501+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:18.629 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:18.501+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:18.629 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:18.501+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:18.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:18.629+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:18.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:18.629+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:18.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:18 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:18.629+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:19.602 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:19 smithi146 podman[78749]: 2024-03-24 12:04:19.326666272 +0000 UTC m=+5.554761294 container died 3ce00834b2066ee47191ad813771ad98ad0097a962f8a64ef1a51bd4c8d3699f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T12:04:19.602 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:19 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:19.508+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:19.602 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:19 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:19.508+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:19.602 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:19 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:19.508+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:19.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:19 smithi146 podman[78749]: 2024-03-24 12:04:19.721062865 +0000 UTC m=+5.949157901 container cleanup 3ce00834b2066ee47191ad813771ad98ad0097a962f8a64ef1a51bd4c8d3699f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:04:19.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:19 smithi146 bash[78749]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5 2024-03-24T12:04:19.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:19 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:19.657+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:19.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:19 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:19.657+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:19.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:19 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:19.657+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:20.288 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:19 smithi146 podman[78777]: 2024-03-24 12:04:19.96967217 +0000 UTC m=+0.643514067 container remove 3ce00834b2066ee47191ad813771ad98ad0097a962f8a64ef1a51bd4c8d3699f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T12:04:20.621 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:20 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:20.486+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:20.622 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:20 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:20.487+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:20.622 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:20 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:20.487+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:20.622 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:20 smithi146 podman[78868]: 2024-03-24 12:04:20.192024683 +0000 UTC m=+0.017506057 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:04:20.622 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:20 smithi146 podman[78868]: 2024-03-24 12:04:20.30374351 +0000 UTC m=+0.129224887 container create 602c0c19fe7727eebdf8869223ce7e33998693a98d90ab27c9ca1ce4b87c6beb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T12:04:20.622 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:20 smithi146 podman[78868]: 2024-03-24 12:04:20.522587274 +0000 UTC m=+0.348068657 container init 602c0c19fe7727eebdf8869223ce7e33998693a98d90ab27c9ca1ce4b87c6beb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T12:04:20.622 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:20 smithi146 podman[78868]: 2024-03-24 12:04:20.525685605 +0000 UTC m=+0.351166986 container start 602c0c19fe7727eebdf8869223ce7e33998693a98d90ab27c9ca1ce4b87c6beb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T12:04:20.622 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:20 smithi146 podman[78868]: 2024-03-24 12:04:20.622752875 +0000 UTC m=+0.448234268 container attach 602c0c19fe7727eebdf8869223ce7e33998693a98d90ab27c9ca1ce4b87c6beb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T12:04:20.884 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 24 12:04:20 smithi146 podman[78868]: 2024-03-24 12:04:20.81013304 +0000 UTC m=+0.635614407 container died 602c0c19fe7727eebdf8869223ce7e33998693a98d90ab27c9ca1ce4b87c6beb (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T12:04:20.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:20 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:20.630+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:20.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:20 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:20.630+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:20.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:20 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:20.630+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:21.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T12:04:21.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:04:21.486 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T12:04:21.486 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T12:04:21.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.6 2024-03-24T12:04:21.803 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:21.476+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:21.803 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:21.476+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:21.803 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:21.476+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:21.803 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:21 smithi146 systemd[1]: Stopping Ceph osd.6 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:04:21.804 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:21.603+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:21.804 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:21.603+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:21.804 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:21.603+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:21.804 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:21.803+0000 7f7fe3470700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:04:21.804 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:21.803+0000 7f7fe3470700 -1 osd.6 156 *** Got signal Terminated *** 2024-03-24T12:04:21.804 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:21 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:21.803+0000 7f7fe3470700 -1 osd.6 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:04:22.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:22 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:22.577+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:22.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:22 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:22.577+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:22.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:22 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:22.577+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:22.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:22 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:22.516+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:22.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:22 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:22.516+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:22.885 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:22 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:22.516+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:23.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:23 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:23.594+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:23.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:23 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:23.594+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:23.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:23 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:23.594+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:23.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:23 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:23.497+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:23.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:23 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:23.497+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:23.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:23 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:23.497+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:24.883 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:24 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:24.501+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:24.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:24 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:24.501+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:24.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:24 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:24.501+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:24.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:24 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:24.587+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:24.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:24 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:24.587+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:24.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:24 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:24.587+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:24.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:24 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:24.587+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:04:01.050378+0000 front 2024-03-24T12:04:01.050384+0000 (oldest deadline 2024-03-24T12:04:24.450532+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:25.630+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:25.630+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:25.630+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:25.630+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:04:01.050378+0000 front 2024-03-24T12:04:01.050384+0000 (oldest deadline 2024-03-24T12:04:24.450532+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:25.460+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:25.460+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:25.460+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:25.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:25 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:25.460+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:26.659+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:36.146569+0000 front 2024-03-24T12:03:36.146684+0000 (oldest deadline 2024-03-24T12:04:00.846611+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:26.659+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:45.948412+0000 front 2024-03-24T12:03:45.948429+0000 (oldest deadline 2024-03-24T12:04:11.248224+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:26.659+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:51.248793+0000 front 2024-03-24T12:03:51.248807+0000 (oldest deadline 2024-03-24T12:04:15.849129+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6[62490]: 2024-03-24T12:04:26.659+0000 7f7fdba5c700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:04:01.050378+0000 front 2024-03-24T12:04:01.050384+0000 (oldest deadline 2024-03-24T12:04:24.450532+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:26.454+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:26.454+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:26.454+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:26.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:26 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:26.454+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:27.629 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:27 smithi146 podman[78973]: 2024-03-24 12:04:27.363204164 +0000 UTC m=+5.660398510 container died 8bab439aede48a17390bd800de536d514ef88607f0c473d496f23b4e2e517317 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T12:04:27.629 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:27 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:27.473+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:27.629 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:27 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:27.473+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:27.629 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:27 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:27.473+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:27.629 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:27 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:27.473+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:27.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:27 smithi146 podman[78973]: 2024-03-24 12:04:27.745481234 +0000 UTC m=+6.042675580 container cleanup 8bab439aede48a17390bd800de536d514ef88607f0c473d496f23b4e2e517317 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:04:27.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:27 smithi146 bash[78973]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6 2024-03-24T12:04:28.309 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:27 smithi146 podman[78989]: 2024-03-24 12:04:27.996543136 +0000 UTC m=+0.634785177 container remove 8bab439aede48a17390bd800de536d514ef88607f0c473d496f23b4e2e517317 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T12:04:28.574 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:28 smithi146 podman[79081]: 2024-03-24 12:04:28.213320385 +0000 UTC m=+0.017470182 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:04:28.574 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:28 smithi146 podman[79081]: 2024-03-24 12:04:28.322233528 +0000 UTC m=+0.126383363 container create ed426ab2ecfd63eab8e0547ee669b48e0bf0d763dac131616bce72edc3204606 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T12:04:28.574 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:28 smithi146 podman[79081]: 2024-03-24 12:04:28.575120144 +0000 UTC m=+0.379269934 container init ed426ab2ecfd63eab8e0547ee669b48e0bf0d763dac131616bce72edc3204606 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:04:28.574 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:28 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:28.487+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:28.574 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:28 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:28.487+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:28.574 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:28 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:28.487+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:28.575 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:28 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:28.487+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:28.861 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:28 smithi146 podman[79081]: 2024-03-24 12:04:28.578436997 +0000 UTC m=+0.382586789 container start ed426ab2ecfd63eab8e0547ee669b48e0bf0d763dac131616bce72edc3204606 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T12:04:28.861 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:28 smithi146 podman[79081]: 2024-03-24 12:04:28.666560167 +0000 UTC m=+0.470709971 container attach ed426ab2ecfd63eab8e0547ee669b48e0bf0d763dac131616bce72edc3204606 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0) 2024-03-24T12:04:29.134 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 24 12:04:28 smithi146 podman[79081]: 2024-03-24 12:04:28.862872569 +0000 UTC m=+0.667022369 container died ed426ab2ecfd63eab8e0547ee669b48e0bf0d763dac131616bce72edc3204606 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T12:04:29.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T12:04:29.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:04:29.473 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T12:04:29.473 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T12:04:29.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.7 2024-03-24T12:04:29.796 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:29.515+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:29.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:29.515+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:29.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:29.515+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:29.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:29.515+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:29.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 systemd[1]: Stopping Ceph osd.7 for 0ddfea52-e9d3-11ee-95cd-87774f69a715... 2024-03-24T12:04:29.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:29.797+0000 7f23e675b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:04:29.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:29.797+0000 7f23e675b700 -1 osd.7 156 *** Got signal Terminated *** 2024-03-24T12:04:29.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:29 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:29.797+0000 7f23e675b700 -1 osd.7 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:04:30.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:30 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:30.505+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:30.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:30 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:30.505+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:30.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:30 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:30.505+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:30.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:30 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:30.505+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:31.633 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:31 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:31.499+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:31.634 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:31 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:31.499+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:31.634 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:31 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:31.499+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:31.634 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:31 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:31.499+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:32.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:32 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:32.468+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:32.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:32 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:32.468+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:32.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:32 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:32.468+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:32.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:32 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:32.468+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:33.883 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:33 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:33.432+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:33.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:33 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:33.432+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:33.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:33 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:33.432+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:33.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:33 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:33.432+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:33.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:33 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:33.432+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.146:6804 osd.4 since back 2024-03-24T12:04:07.832792+0000 front 2024-03-24T12:04:07.832839+0000 (oldest deadline 2024-03-24T12:04:33.132852+0000) 2024-03-24T12:04:34.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:34 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:34.445+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-03-24T12:03:35.129033+0000 front 2024-03-24T12:03:35.129108+0000 (oldest deadline 2024-03-24T12:04:03.329442+0000) 2024-03-24T12:04:34.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:34 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:34.445+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-03-24T12:03:43.329983+0000 front 2024-03-24T12:03:43.330079+0000 (oldest deadline 2024-03-24T12:04:08.029806+0000) 2024-03-24T12:04:34.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:34 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:34.445+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-03-24T12:03:52.631256+0000 front 2024-03-24T12:03:52.631178+0000 (oldest deadline 2024-03-24T12:04:17.931062+0000) 2024-03-24T12:04:34.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:34 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:34.445+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-03-24T12:03:59.032067+0000 front 2024-03-24T12:03:59.032112+0000 (oldest deadline 2024-03-24T12:04:24.931989+0000) 2024-03-24T12:04:34.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:34 smithi146 ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7[64743]: 2024-03-24T12:04:34.445+0000 7f23e2d4f700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.146:6804 osd.4 since back 2024-03-24T12:04:07.832792+0000 front 2024-03-24T12:04:07.832839+0000 (oldest deadline 2024-03-24T12:04:33.132852+0000) 2024-03-24T12:04:35.488 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:35 smithi146 podman[79189]: 2024-03-24 12:04:35.220519555 +0000 UTC m=+5.527424342 container died 85b15c0b61146be19720f32b7640204ab4e3ba8695b07e37d9a7ff6f5749ceb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T12:04:35.839 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:35 smithi146 podman[79189]: 2024-03-24 12:04:35.664304864 +0000 UTC m=+5.971209682 container cleanup 85b15c0b61146be19720f32b7640204ab4e3ba8695b07e37d9a7ff6f5749ceb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-24T12:04:35.839 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:35 smithi146 bash[79189]: ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7 2024-03-24T12:04:36.134 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:35 smithi146 podman[79252]: 2024-03-24 12:04:35.840548815 +0000 UTC m=+0.620924584 container remove 85b15c0b61146be19720f32b7640204ab4e3ba8695b07e37d9a7ff6f5749ceb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:04:36.542 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:36 smithi146 podman[79341]: 2024-03-24 12:04:36.156709132 +0000 UTC m=+0.017984323 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:04:36.542 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:36 smithi146 podman[79341]: 2024-03-24 12:04:36.257982069 +0000 UTC m=+0.119257242 container create a588132e8cf3d92b8dfc6c814aae67c66b4bc21f77ac3db49c44f6a58d89397c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T12:04:36.542 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:36 smithi146 podman[79341]: 2024-03-24 12:04:36.468835661 +0000 UTC m=+0.330110842 container init a588132e8cf3d92b8dfc6c814aae67c66b4bc21f77ac3db49c44f6a58d89397c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T12:04:36.542 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:36 smithi146 podman[79341]: 2024-03-24 12:04:36.472143957 +0000 UTC m=+0.333419134 container start a588132e8cf3d92b8dfc6c814aae67c66b4bc21f77ac3db49c44f6a58d89397c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T12:04:36.543 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:36 smithi146 podman[79341]: 2024-03-24 12:04:36.54358401 +0000 UTC m=+0.404859195 container attach a588132e8cf3d92b8dfc6c814aae67c66b4bc21f77ac3db49c44f6a58d89397c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T12:04:36.884 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 24 12:04:36 smithi146 podman[79341]: 2024-03-24 12:04:36.756176117 +0000 UTC m=+0.617451290 container died a588132e8cf3d92b8dfc6c814aae67c66b4bc21f77ac3db49c44f6a58d89397c (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T12:04:37.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ddfea52-e9d3-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T12:04:37.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:04:37.544 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T12:04:37.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:07:38.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:08:00.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:08:00.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:08:00.098 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T12:08:00.100 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932/remote/smithi032/crash 2024-03-24T12:08:00.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:08:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:08:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:08:00.162 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932/remote/smithi146/crash 2024-03-24T12:08:00.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:08:00.204 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:08:00.204 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:08:00.205 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T12:08:00.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T12:08:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:2024-03-24T11:44:15.457063+0000 mon.smithi032 (mon.0) 514 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:08:00.236 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T12:08:00.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[SEC\]' /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T12:08:00.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]' /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T12:08:00.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[WRN\]' /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T12:08:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:2024-03-24T11:44:15.457063+0000 mon.smithi032 (mon.0) 514 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:08:00.446 INFO:tasks.cephadm:Compressing logs... 2024-03-24T12:08:00.447 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:08:00.488 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:08:00.513 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-24T12:08:00.513 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T12:08:00.514 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mgr.smithi032.xjdgbw.log 2024-03-24T12:08:00.514 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:08:00.514 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T12:08:00.514 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mgr.smithi032.xjdgbw.log: /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T12:08:00.516 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.1.log: -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi032.qzdcjy.log 2024-03-24T12:08:00.516 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T12:08:00.516 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi032.qzdcjy.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mon.smithi032.log 2024-03-24T12:08:00.516 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:08:00.516 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi032.qzdcjy.log.gz 2024-03-24T12:08:00.517 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-03-24T12:08:00.517 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/cephadm.log 2024-03-24T12:08:00.517 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T12:08:00.517 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:08:00.518 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.4.log: /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mon.smithi146.log 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mon.smithi146.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mgr.smithi146.tfohfl.log 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:08:00.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mgr.smithi146.tfohfl.log: 91.6% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:08:00.522 INFO:teuthology.orchestra.run.smithi146.stderr: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:08:00.523 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mon.smithi032.log: 91.5% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:08:00.523 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:08:00.523 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T12:08:00.523 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log: 85.7% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:08:00.524 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:08:00.525 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:08:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.2.log: /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-volume.log: 89.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:08:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi146.xsrsue.log 2024-03-24T12:08:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:08:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi146.xsrsue.log: 87.8% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi146.xsrsue.log.gz 2024-03-24T12:08:00.541 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:08:00.543 INFO:teuthology.orchestra.run.smithi032.stderr: 91.4% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:08:00.566 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-volume.log: 91.4% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:08:00.582 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mgr.smithi146.tfohfl.log.gz 2024-03-24T12:08:00.845 INFO:teuthology.orchestra.run.smithi146.stderr: 92.6% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mon.smithi146.log.gz 2024-03-24T12:08:00.915 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mgr.smithi032.xjdgbw.log.gz 2024-03-24T12:08:02.012 INFO:teuthology.orchestra.run.smithi032.stderr: 91.9% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-mon.smithi032.log.gz 2024-03-24T12:08:13.461 INFO:teuthology.orchestra.run.smithi032.stderr: 94.4% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T12:08:15.412 INFO:teuthology.orchestra.run.smithi146.stderr: 93.9% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T12:08:17.015 INFO:teuthology.orchestra.run.smithi146.stderr: 94.2% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T12:08:18.726 INFO:teuthology.orchestra.run.smithi032.stderr: 94.1% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T12:08:19.750 INFO:teuthology.orchestra.run.smithi032.stderr: 94.2% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T12:08:19.755 INFO:teuthology.orchestra.run.smithi146.stderr: 94.1% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T12:08:20.401 INFO:teuthology.orchestra.run.smithi032.stderr: 94.1% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T12:08:20.403 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-24T12:08:20.403 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m19.902s 2024-03-24T12:08:20.403 INFO:teuthology.orchestra.run.smithi032.stderr:user 1m9.574s 2024-03-24T12:08:20.403 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m2.542s 2024-03-24T12:08:25.766 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/0ddfea52-e9d3-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T12:08:25.768 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-24T12:08:25.768 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m25.265s 2024-03-24T12:08:25.768 INFO:teuthology.orchestra.run.smithi146.stderr:user 1m13.233s 2024-03-24T12:08:25.768 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m2.882s 2024-03-24T12:08:25.768 INFO:tasks.cephadm:Archiving logs... 2024-03-24T12:08:25.768 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932/remote/smithi032/log 2024-03-24T12:08:25.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:08:34.792 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932/remote/smithi146/log 2024-03-24T12:08:34.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:08:45.194 INFO:tasks.cephadm:Removing cluster... 2024-03-24T12:08:45.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 --force 2024-03-24T12:08:45.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ddfea52-e9d3-11ee-95cd-87774f69a715 --force 2024-03-24T12:08:45.956 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T12:08:45.956 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T12:08:45.971 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T12:08:45.986 INFO:tasks.cephadm:Teardown complete 2024-03-24T12:08:45.986 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T12:08:45.996 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T12:08:45.996 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:08:46.013 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:08:46.025 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:08:46.029 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:08:46.029 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-03-24T12:08:46.029 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +873us[ +873us] +/- 76ms 2024-03-24T12:08:46.029 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 +120us[ +179us] +/- 56ms 2024-03-24T12:08:46.029 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -780us[ -780us] +/- 64ms 2024-03-24T12:08:46.030 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:08:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:08:46.044 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:08:46.045 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-24T12:08:46.045 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +867us[ +867us] +/- 76ms 2024-03-24T12:08:46.045 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 +96us[ +145us] +/- 56ms 2024-03-24T12:08:46.045 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -798us[ -798us] +/- 64ms 2024-03-24T12:08:46.045 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:08:46.045 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T12:08:46.054 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T12:08:46.055 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T12:08:46.064 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:08:46.155 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:08:46.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:08:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:08:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:08:46.292 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T12:08:46.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T12:08:46.311 INFO:teuthology.task.internal:Duration was 2062.065018 seconds 2024-03-24T12:08:46.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T12:08:46.320 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T12:08:46.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:08:46.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:08:46.362 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:08:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:08:46.749 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T12:08:46.750 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-03-24T12:08:46.750 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:08:46.809 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-03-24T12:08:46.810 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:08:46.836 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T12:08:46.836 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:08:46.852 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:08:46.904 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T12:08:46.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:08:46.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:09:01.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T12:09:01.378 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T12:09:01.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:09:01.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:09:01.436 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T12:09:01.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:09:01.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:09:01.472 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-03-24T12:09:01.505 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-03-24T12:09:01.582 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:09:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.596 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:09:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:09:01.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T12:09:01.648 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T12:09:01.649 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932/remote/smithi032 2024-03-24T12:09:01.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:09:01.709 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932/remote/smithi146 2024-03-24T12:09:01.709 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:09:01.752 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T12:09:01.752 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:09:01.754 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:09:01.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T12:09:01.811 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T12:09:01.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T12:09:01.820 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T12:09:01.820 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:09:01.822 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:09:01.834 INFO:teuthology.orchestra.run.smithi032.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:09 /home/ubuntu/cephtest 2024-03-24T12:09:01.854 INFO:teuthology.orchestra.run.smithi146.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:09 /home/ubuntu/cephtest 2024-03-24T12:09:01.855 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T12:09:01.865 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T12:09:01.886 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T12:09:01.911 DEBUG:teuthology.nuke:shortname: smithi032 2024-03-24T12:09:01.912 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:09:01.933 DEBUG:teuthology.nuke:shortname: smithi146 2024-03-24T12:09:01.934 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:09:01.936 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:25:48.641389', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHla/g2BW76tkdDgkUHtnGhcCbXVqvUfdl8sdVsX4H8tQ9mLhsI+DMVJ83PchBBkU6L3/6wFg/YOksAlUPPue4U='} 2024-03-24T12:09:01.958 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617932', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:25:48.639381', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvGKTTtmLi+nEB3wCQRY0dsYqc9wyxZ4qLeJN2bz1MuKLHAsrAKaJuCChldvnIbWjUoTKB0/z/RtxeUxC+Sgkc='} 2024-03-24T12:09:01.983 INFO:teuthology.orchestra.console.smithi032:Power off 2024-03-24T12:09:01.983 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:09:02.009 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-03-24T12:09:02.009 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:09:02.022 INFO:teuthology.orchestra.console.smithi146:Power off 2024-03-24T12:09:02.023 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:09:02.034 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-24T12:09:02.045 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-03-24T12:09:02.045 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:09:02.068 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-24T12:09:06.035 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:09:06.069 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:09:06.146 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-24T12:09:06.178 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-24T12:09:10.150 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:09:10.179 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:09:10.260 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-03-24T12:09:10.289 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-03-24T12:09:10.362 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-03-24T12:09:10.390 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-03-24T12:09:10.494 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2062.065017938614 failure_reason: '"2024-03-24T11:44:15.457063+0000 mon.smithi032 (mon.0) 514 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-24T12:09:10.495 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T12:09:10.570 INFO:teuthology.run:FAIL